Tuesday, October 1, 2019

GIT : Commands and Operation

 Commands and Operation



Remote Master-------------------------------------Remote branch
|                                                                         |
|                                                                         |
|                                                                         |
|                                                                         |
Local Master ---------------------------------------Local branch


Usual Operations :

  1. Usually project will be made available in Git
  2. User clones the repository into local (once you clone master you get all the branches in the repository)
  3. creates a story branch locally if not available / if available checkout that branch
  4. Works on development work  
  5. commits the Changes to local branch
  6. Pushes the changes to remote branch
  7. Then the code pushed code is merged to master in remote.

Scenario 1: Developer wants to use the new code his colleague merged recently to master.
  1. Saves current changes in his current branch
  2. Developer switches to local master
  3. Perform PULL request to update his local master with latest changes.
  4. checkout the story branch
  5. Merges the updated the master to local branch
Scenario 2: Developer completed his coding while merging his code to remote master , he faces merge conflicts.
  1. checkout master
  2. pull all the changes from remote master to local master
  3. checkout story branch
  4. perform merge 
  5. Solve all the conflicts 
    1. Git shows "AUTO CONFLICT merge xxxx"
    2. Files turn red (Eg :intelliJ)
    3. rt ck > Git > resolve conflicts
    4. click Merge button
    5. Window with 3 screens open : middle screen - Current code , left and right screen shows the code in local staging and code in master
    6. Make required selection
    7. Save
  6. Commit changes
  7. Push to story branch
  8. Raise PR (merge to master) 
Sample Code :
git checkout master
git pull
git checkout my_branch
git merge master
(now use above conflict resolution to solve the issue )

Clone-Remote Master into local:
Usually any user will either clone the remote repository into his computer or create a local master repository.
git clone git_ssh_link #clone from git
git init #new repository without cloning

Check status: 
git branch

Result:
*master

Create / Use already existing branch 
once user has cloned / create a local repository
git checkout - b branch_new #Create a new branch in local and switch
git  branch branch_new #Just create a new branch in local

Message display / status
git branch #display all branches
git branch -vv #display all branches with their upstream branch
git status #display status of project wrt git
git log

Change branch 
git checkout branch_new #switch branch

Commit changes to staging whenever developer has reached a checkpoint
git add file #add file to local staging
git commit -m "description" #commit added file in staging

Push changes to story branch in remote (cloud)
git add file #add file to local staging
git commit -m "description" #commit added file in staging
git push -u origin branch_new #Create new story branch in remote and put changes there

Update the current branch if new changes are available in remote

git pull origin branch_name #1st time only
git pull #get latest updates from remote branch whichever is made upstream else same branch which we are currenlty

Remove the file from local git
git rm -rf file_name #delete file from local git

Reset
git checkout file #reset changes of the current file
git reset #save and reset changes of all the files to last commit
git reset --hard #reset to last commit without commiting

Delete branch
git branch -D branch-name

To Create a upstream
git branch -u masterbranchname # from where u want to pull latest code


Example to create a new project locally and push it to remote :
  • git init
  • git add README.md
  • git commit -m "first commit"
  • git remote add origin git@github.xxx.com:username/scala_protobuf.git
  • git push -u origin master

…or push an existing repository from the command line

  • git remote add origin git@github.xxx.com:username/scala_protobuf.git
  • git push -u origin master

Git Remove File from Face of the earth (all history included)
git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch path_to_file" HEAD
git push --all

Git delete remote branch
git push origin --delete branch_name


Wednesday, September 18, 2019

Linux / Mac : Get latest of chromeDriver

Linux / Mac : Get latest of chromeDriver



LINUX
currdir=$(pwd)/misc
latest_version=$(wget https://chromedriver.storage.googleapis.com/LATEST_RELEASE -O -) && wget https://chromedriver.storage.googleapis.com/${latest_version}/chromedriver_linux64.zip -P $currdir
unzip -q $(pwd)/misc/chromedriver_mac64.zip && /bin/rm $(pwd)/misc/chromedriver_linux64.zip


MAC
#mac make sure "wget" is installed
#brew install wget
currdir=$(pwd)/misc
latest_version=$(wget https://chromedriver.storage.googleapis.com/LATEST_RELEASE -O -) && wget https://chromedriver.storage.googleapis.com/${latest_version}/chromedriver_mac64.zip -P $currdir
unzip -q $(pwd)/misc/chromedriver_mac64.zip && /bin/rm $(pwd)/misc/chromedriver_mac64.zip

Tuesday, September 17, 2019

scala : Perform basic test using MOCK


Scala : Perform basic test using MOCK and scalatest


Note :
  1. If you already have scala installed in your system do not add in your scalaVersion := "2.13.0" build file.
  2. Always add code in the test folder to use mock and scalatest.

build.sbt (I already have scala in my PC hence did not add scalaversion)

name := "sbt_test" version := "0.1"
//scalaVersion := "2.13.1"

libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.0" libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.4" % "test" libraryDependencies += "org.mockito" % "mockito-all" % "1.8.4" libraryDependencies += "org.scalamock" %% "scalamock" % "4.3.0" % "test" libraryDependencies += "org.testng" % "testng" % "6.10" 

 Under Project > src > test > scala > test1 > test_class.scala
package test1
import org.scalatest.{BeforeAndAfter, FunSpec, FunSuite}
import org.scalamock.scalatest.MockFactory

object test_class extends App with MockFactory{
 val x=mock[Point]
}

class Point(xc: Int, yc: Int) {
  var x: Int = xc
  var y: Int = yc

  def start(x:Int,y:Int): Unit ={
    println("start")
  }

  def move(dx: Int, dy: Int) {
    println("move")
  }

  def end(dx: Int) {
    println("end")
  }
}

Bazel : Debugging / JVM debugging using JDB for Scala - Bazel


Debugging / JVM debugging using JDB


Follow steps to perform debugging in Bazel / Any JVMs :

Steps :
1.cd path
2. Open new Terminal Window
3. bazel run build_name --java_debug
4. Open another Terminal Window
5. jdb -attach localhost:5005 -sourcepath ~/dev/remote/src/main/java/

Note :Breakpoint can only be set in Dev Code not on scalatest .

Eg:
#stop in Dev code
stop in dev_package.class_name.method

#stop at Dev Code
stop at dev_package.class_name:130

#Clear BreakPoint
clear dev_package.class_name:line_number
clear dev_package.class_name.method

Commands :
run
cont
step over
next
locals
print resultdf.show()


Ref : http://pages.cs.wisc.edu/~horwitz/jdb/jdb.html

Friday, September 13, 2019

Git : Setting up a clone project


Git Pre- Req

Computer Authentication :


Pre- Req:

Windows:

                         i.         Open https://git-scm.com/download/win
                       ii.         Install (Next > next >…)
                      iii.         Right click on select git bash option

Mac :

·       Open Terminal

Steps :


1.     Enter ssh-keygen -t rsa in your git bash prompt / terminal (type “yes” when asked and press enter key otherwise)
2.     Note down location of the file created or usually file gets created in below locations
3.     Open id_rsa.pub file
a.     For Windows :   C:\User_name\.ssh\id_rsa.pub or  C:\xxx_USER \.ssh\id_rsa.pub
b.     For Mac :          /Users/User_name/.ssh/id_rsa.pub

4.     Copy entire Contents
Eg :
ssh-rsa sadfasfldfllflamflamsflWWQEW;as
--------
--------
---------------------------------@in.xxx.com
5.    Create a new git hub account
6.     Goto Settings
7.     Click on SSH and GPG keys
8.     Click on New SSH Keys
9.     Paste the contents inside
10.  Click on Add button
11.  Create a new folder in your system with name git_test (any location)
12.  Navigate inside the folder
13.   
a.     For windows : Open git bash by right click
b.     For mac : use terminal to navigate inside the new folder using “cd command”

14.  In your terminal / git bash run command (copy link below from the git project online)
15.  Run command “cd” into the cloned folder

Monday, September 9, 2019

scala : Basic Setup for Scala and Spark

Basic Setup for Scala

Scala can be run using :
  1. Spark-Shell
  2. SBT

Spark-Shell :Usually this is used as it provides scala along with spark capabilites to create RDD/DataSet and DF

Installation for Linux
  1. java -version
  2. sudo apt-get install scala #scala -version
    1. type $scala
    2. println("hi")
    3. :q
  3. goto - https://spark.apache.org/downloads.html
  4. Download latest tar file
  5. tar -zxvf spark-2.0.2-bin-hadoop2.7.tgz
  6. cd spark-2.0.2-bin-hadoop2.7.tgz
    1. ./spark-shell
    2. :q //to quit
    3. open .bashrc
  7. add line SPARK_HOME=~/Downloads/spark-3.0.0-preview2-bin-hadoop2.7
  8. $export PATH=$SPARK_HOME/bin:$PATH
  9. $source ~/.bashrc
  10. $spark-shell
Installation for mac
  1. Run homebrew installer : $/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"#(visit homebrew for more details)
  2. $xcode-select –install
  3. $brew cask install java
  4. $brew install scala
  5. $brew install apache-spark
  6. $spark-shell # to start spark-shell
  7. $brew upgrade apache-spark #to Upgrade 
Start session 
$spark-shell

To Create a Dataframe:
:imports
#copy 4th point as 
#import org.apache.spark.sql.functions._ as  
#import org.apache.spark.sql._
import org.apache.spark.sql._
val mockDF1: DataFrame = Seq((0, "A"), (1, "B"), (0, "C")).toDF("col1", "col2")
mockDF1

TO uninstall 
brew uninstall scala
--------------

SBT shell

Linux
  • Install brew
  • open terminal
  • Enter "/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
  • After brew is installed successfully
  • Enter"brew install sbt@1"
  • TO uninstall - brew uninstall scala
Linux :

Getting SBT Started in Terminal:
  1. Open Terminal
  2. enter "sbt"
  3. After sbt shell is opened
  4. Open Scala REPL session inside SBT using - "console" or "consoleQuick"
  5. Type "println("helloworld")
  6. To quit type ":q" or":quit"
  7. And "exit" to exit sbt shell
Note : You can only do basic operations here . But cannot do operations where there is dependency on libraries .For that you need a build tool like sbt or bazel .

To add library into your sbt shell :
  1. Download required jar file Eg:https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.12/2.4.4
  2. Open sbt shell
  3. run cmd ":require spark-sql_2.12.jar" #Added - file.jar
import org.apache.spark.sql.{Column, DataFrame, SaveMode, SparkSession}

Using SBT in IntelliJ :
  1. IntelliJ by default comes with sbt
  2. Install Idea intelliJ
  3. Create a new sbt project
  4. goto project /src/main/test
  5. Create a new file "test1.scala"
  6. copy below code  
  7. set SBT SDT to 2.11
  8. add contents into build file
  9. Run it (rt ck run)
build.sbt:

name := "sbt_test" version := "0.1" scalaVersion := "2.11.8"
libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % "2.1.0", "org.apache.spark" %% "spark-sql" % "2.1.0")

libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.0"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.4" % "test"
libraryDependencies += "org.mockito" % "mockito-all" % "1.8.4"
libraryDependencies += "org.scalamock" %% "scalamock" % "4.3.0" % "test"
libraryDependencies += "org.testng" % "testng" % "6.10"

//Basic Class:

object test1 extends App{
  println("Hello World")
}
or 
object test1 { def
            main(args:
              Array[String]): Unit = println("Hello, World!") }
or

import org.apache.spark.sql.SparkSession
import org.scalamock.scalatest.MockFactory
import org.apache.spark.sql.{Column, DataFrame, SaveMode, SparkSession}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._

object test1  extends App with MockFactory {
  val spark = SparkSession.builder().master("local").appName("spark-shell").getOrCreate()
  import spark.implicits._
  val df1: DataFrame = Seq((1,1), (2,2), (3,3)).toDF("col1", "col2")
  df1.show()
}

Issues :

1. Error:scalac: Multiple 'scala-library*.jar' files (scala-library.jar, scala-library.jar, scala-library.jar) in
          Scala
            compiler classpath in
          Scala
            SDK scala-sdk-2.11.7`

Solution:
File > Project_Structure > Libraries     
Remove "SBT:org.scala-lang:scala-library:2.11.8:jar"

2. Cannot resolve App
Solution:
Set the current Scala SDT to 2.11.8
--------------------------------------------------------------------------------------------------------
Issues :

1. Error:scalac: Multiple 'scala-library*.jar' files (scala-library.jar, scala-library.jar, scala-library.jar) in
        Scala
          compiler classpath in
        Scala
          SDK scala-sdk-2.11.7`

Solution : Goto File > Project Stucture > Platform Settings > SDKs > Remove Duplicate
Solution 2: Remove "scalaVersion := "2.xx.xx" from build.sbt file
Solution 3:


2. Could not find or load main class in scala in intellij IDE
Solution: Right click on "src folder" and select Mark Directory as -> Sources root
--------------------------------------------------------------------------------------------------------

Ref: