Linux / Mac : Get latest of chromeDriver

Linux / Mac : Get latest of chromeDriver

LINUX
currdir=$(pwd)/misc
latest_version=$(wget https://chromedriver.storage.googleapis.com/LATEST_RELEASE -O -) && wget https://chromedriver.storage.googleapis.com/${latest_version}/chromedriver_linux64.zip -P $currdir

unzip -q $(pwd)/misc/chromedriver_mac64.zip && /bin/rm $(pwd)/misc/chromedriver_linux64.zip

MAC
#mac make sure "wget" is installed
#brew install wget
currdir=$(pwd)/misc

latest_version=$(wget https://chromedriver.storage.googleapis.com/LATEST_RELEASE -O -) && wget https://chromedriver.storage.googleapis.com/${latest_version}/chromedriver_mac64.zip -P $currdir
unzip -q $(pwd)/misc/chromedriver_mac64.zip && /bin/rm $(pwd)/misc/chromedriver_mac64.zip

scala : Perform basic test using MOCK

Scala : Perform basic test using MOCK and scalatest

Note :

1. If you already have scala installed in your system do not add in your scalaVersion := "2.13.0" build file.
2. Always add code in the test folder to use mock and scalatest.

build.sbt (I already have scala in my PC hence did not add scalaversion)

name := "sbt_test" version := "0.1"

//scalaVersion := "2.13.1"

libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.0" libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.4" % "test" libraryDependencies += "org.mockito" % "mockito-all" % "1.8.4" libraryDependencies += "org.scalamock" %% "scalamock" % "4.3.0" % "test" libraryDependencies += "org.testng" % "testng" % "6.10" 

 Under Project > src > test > scala > test1 > test_class.scala

package test1

import org.scalatest.{BeforeAndAfter, FunSpec, FunSuite}

import org.scalamock.scalatest.MockFactory

object test_class extends App with MockFactory{

 val x=mock[Point]

}

class Point(xc: Int, yc: Int) {

  var x: Int = xc

  var y: Int = yc

  def start(x:Int,y:Int): Unit ={

    println("start")

  }

  def move(dx: Int, dy: Int) {

    println("move")

  }

  def end(dx: Int) {

    println("end")

  }

}

Bazel : Debugging / JVM debugging using JDB for Scala - Bazel

Debugging / JVM debugging using JDB

Follow steps to perform debugging in Bazel / Any JVMs :

Steps :
1.cd path
2. Open new Terminal Window
3. bazel run build_name --java_debug
4. Open another Terminal Window
5. jdb -attach localhost:5005 -sourcepath ~/dev/remote/src/main/java/

Note :Breakpoint can only be set in Dev Code not on scalatest .

Eg:
#stop in Dev code
stop in dev_package.class_name.method

#stop at Dev Code
stop at dev_package.class_name:130

#Clear BreakPoint
clear dev_package.class_name:line_number
clear dev_package.class_name.method

Commands :
run
cont
step over
next
locals
print resultdf.show()


Ref : http://pages.cs.wisc.edu/~horwitz/jdb/jdb.html

Git : Setting up a clone project

Git Pre- Req

Computer Authentication :

Pre- Req:

Windows:

                         i.         Open https://git-scm.com/download/win

                       ii.         Install (Next > next >…)

                      iii.         Right click on select git bash option

Mac :

·       Open Terminal

Steps :

1.     Enter ssh-keygen -t rsa in your git bash prompt / terminal (type “yes” when asked and press enter key otherwise)

2.     Note down location of the file created or usually file gets created in below locations

3.     Open id_rsa.pub file

a.     For Windows :   C:\User_name\.ssh\id_rsa.pub or  C:\xxx_USER \.ssh\id_rsa.pub

b.     For Mac :          /Users/User_name/.ssh/id_rsa.pub

4.     Copy entire Contents

Eg :

ssh-rsa sadfasfldfllflamflamsflWWQEW;as

--------

--------

---------------------------------@in.xxx.com

5.    Create a new git hub account

6.     Goto Settings

7.     Click on SSH and GPG keys

8.     Click on New SSH Keys

9.     Paste the contents inside

10.  Click on Add button

11.  Create a new folder in your system with name git_test (any location)

12.  Navigate inside the folder

13.   

a.     For windows : Open git bash by right click

b.     For mac : use terminal to navigate inside the new folder using “cd command”

14.  In your terminal / git bash run command (copy link below from the git project online)

“ git clone git@github.xxx.com:djayapr1/testing.git”

15.  Run command “cd” into the cloned folder

scala : Basic Setup for Scala and Spark

Basic Setup for Scala

Scala can be run using :

1. Spark-Shell
2. SBT

Spark-Shell :Usually this is used as it provides scala along with spark capabilites to create RDD/DataSet and DF

Installation for Linux

1. java -version
2. sudo apt-get install scala #scala -version
1. type $scala
2. println("hi")
3. :q
3. goto - https://spark.apache.org/downloads.html
4. Download latest tar file
5. tar -zxvf spark-2.0.2-bin-hadoop2.7.tgz
6. cd spark-2.0.2-bin-hadoop2.7.tgz
1. ./spark-shell
2. :q //to quit
3. open .bashrc
7. add line SPARK_HOME=~/Downloads/spark-3.0.0-preview2-bin-hadoop2.7
8. $export PATH=$SPARK_HOME/bin:$PATH
9. $source ~/.bashrc
10. $spark-shell

Installation for mac

1. Run homebrew installer : $/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"#(visit homebrew for more details)
2. $xcode-select –install
3. $brew cask install java
4. $brew install scala
5. $brew install apache-spark
6. $spark-shell # to start spark-shell
7. $brew upgrade apache-spark #to Upgrade 

Start session 
$spark-shell

To Create a Dataframe:
:imports
#copy 4th point as 

#import org.apache.spark.sql.functions._ as  

#import org.apache.spark.sql._

import org.apache.spark.sql._

val mockDF1: DataFrame = Seq((0, "A"), (1, "B"), (0, "C")).toDF("col1", "col2")

mockDF1

TO uninstall 

brew uninstall scala

--------------

SBT shell

Linux

• Install brew
• open terminal
• Enter "/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
• After brew is installed successfully
• Enter"brew install sbt@1"
• TO uninstall - brew uninstall scala

Linux :

Ref : http://www.codebind.com/linux-tutorials/install-scala-sbt-java-ubuntu-18-04-lts-linux/

https://www.scala-sbt.org/1.x/docs/Installing-sbt-on-Linux.html

Getting SBT Started in Terminal:

1. Open Terminal
2. enter "sbt"
3. After sbt shell is opened
4. Open Scala REPL session inside SBT using - "console" or "consoleQuick"
5. Type "println("helloworld")
6. To quit type ":q" or":quit"
7. And "exit" to exit sbt shell

Note : You can only do basic operations here . But cannot do operations where there is dependency on libraries .For that you need a build tool like sbt or bazel .

To add library into your sbt shell :

1. Download required jar file Eg:https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.12/2.4.4
2. Open sbt shell
3. run cmd ":require spark-sql_2.12.jar" #Added - file.jar

import org.apache.spark.sql.{Column, DataFrame, SaveMode, SparkSession}

Using SBT in IntelliJ :

1. IntelliJ by default comes with sbt
2. Install Idea intelliJ
3. Create a new sbt project
4. goto project /src/main/test
5. Create a new file "test1.scala"
6. copy below code  
7. set SBT SDT to 2.11
8. add contents into build file
9. Run it (rt ck run)

build.sbt:

name := "sbt_test" version := "0.1" scalaVersion := "2.11.8"
libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % "2.1.0", "org.apache.spark" %% "spark-sql" % "2.1.0")

libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.0"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.4" % "test"
libraryDependencies += "org.mockito" % "mockito-all" % "1.8.4"
libraryDependencies += "org.scalamock" %% "scalamock" % "4.3.0" % "test"
libraryDependencies += "org.testng" % "testng" % "6.10"

//Basic Class:

object test1 extends App{

  println("Hello World")

}

or 

object test1 { def

            main(args:

              Array[String]): Unit = println("Hello, World!") }

or

import org.apache.spark.sql.SparkSession

import org.scalamock.scalatest.MockFactory

import org.apache.spark.sql.{Column, DataFrame, SaveMode, SparkSession}

import org.apache.spark.sql.functions._

import org.apache.spark.sql.types._

object test1  extends App with MockFactory {

  val spark = SparkSession.builder().master("local").appName("spark-shell").getOrCreate()

  import spark.implicits._

  val df1: DataFrame = Seq((1,1), (2,2), (3,3)).toDF("col1", "col2")

  df1.show()

}

Issues :

1. Error:scalac: Multiple 'scala-library*.jar' files (scala-library.jar, scala-library.jar, scala-library.jar) in

          Scala

            compiler classpath in

          Scala

            SDK scala-sdk-2.11.7`

Solution:

File > Project_Structure > Libraries     

Remove "SBT:org.scala-lang:scala-library:2.11.8:jar"

2. Cannot resolve App

Solution:

Set the current Scala SDT to 2.11.8

--------------------------------------------------------------------------------------------------------

Issues :

1. Error:scalac: Multiple 'scala-library*.jar' files (scala-library.jar, scala-library.jar, scala-library.jar) in

        Scala

          compiler classpath in

        Scala

          SDK scala-sdk-2.11.7`

Solution : Goto File > Project Stucture > Platform Settings > SDKs > Remove Duplicate
Solution 2: Remove "scalaVersion := "2.xx.xx" from build.sbt file
Solution 3:

2. Could not find or load main class in scala in intellij IDE
Solution: Right click on "src folder" and select Mark Directory as -> Sources root

--------------------------------------------------------------------------------------------------------

Ref:

https://www.analyticsvidhya.com/blog/2017/01/scala/