Scala : HTTP Client using Sttp

 HTTP Client using Sttp

Scala :Protobuf

Scala :Protobuf

Protobuf file (Like json and xml , protobuf is used to store and transmit data).

Below are some of the advantages of protobuf

1. Scheme is preserved

2. similar to json / xml

3. Cross platform

4. Parsing is faster

5. Compressed

Example :

Pre - Req : Create a new project in scala using sbt

build.sbt

name := "protobuf"

version := "0.1"

scalaVersion := "2.12.7"

PB.targets in Compile := Seq(

  scalapb.gen() -> (sourceManaged in Compile).value

)

PB.protoSources in Test := Seq(file("/src/protobuf/"))

libraryDependencies += "com.thesamet.scalapb" %% "scalapb-runtime" % scalapb.compiler.Version.scalapbVersion % "protobuf"

project_name/project/plugins.sbt

addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.18")

libraryDependencies += "com.thesamet.scalapb

src/main/protobuf/person.proto

syntax = "proto3";

message Person {

    string name = 1;

    int32 age = 2;

    Gender gender = 3;

}

enum Gender {

    MALE = 0;

    FEMALE = 1;

}

src/main/Persontest.scala

import com.google.protobuf.ByteString

import java.io.{InputStream, OutputStream}

import person.{Gender, Person}

object Persontest {

  val person = Person(

    name = "User1",

    age = 15,

    gender = Gender.MALE

  )

}

Scala : Reflections

Scala : Reflections 

used to Convert the input string into Scala object.
Eg :
In below example class name "testsubclass" is a string in the main object Which is converted into scala object.

class testsubclass { 

def hi:String=return ("newx") 

}

 
object Test{ 

def main((args: Array[String]):{

val x:Any=(Class.forName("testsubclass").newInstance() println(x.asInstanceOf[{def hi:String}]).hi) 

//method declared and method name called should match 

} }

scala : Traits

Traits

Summary :

1. They are similar to interfaces in java
2. They cannot be instantiated
3. Unlike Java they can have methods.
4. Instances of objects can extend trait
5. Uses "with" or "extends keyword

Usages :

1. Wherever usability is a priority
2. When there is no IS-A relation ship ie., unrelated classes
3. When the class under cnsideration already has extended another class (this is very rare as this is usually a design flaw.

Note : Trait can also extend an object instance .

UseCase:

trait Debugger {
def log(message: String) {
do something with message
}}

val problemChild = new ProblemChild ()with Debugger

Example : 

object testing extends App{

  val obj1:SomeClass=new SomeClass() // 

  val x = new newClass() with Trait1 //Object instance extending a trait

  println(x.Tvar1)

  println(x.Cvar3)

  

}

//-------------------

class SomeClass extends newClass with Trait1 with Trait2 {

  println(Tvar1)

  println(Tvar2)

  println(Cvar3)

}

trait Trait1{

  val Tvar1="trait1"}

trait Trait2{

  val Tvar2="trait2"}

class newClass(){

val Cvar3="class"}

 Result :
trait1
trait2
class
trait1
class

Scala : Provide column names as Seq to select statement in Scala

Scala : Provide column names as Seq to select statement in Scala 

Sample Code :

val df1: DataFrame = Seq((1,1,"kjj"), (2,2,"987987")).toDF("col1", "col2","col3")

val columnName = Seq("col1", "col3")

val DFFiltered = df1.select(columnName.map(name => col(name)): _*)

DFFiltered.show()

Full Code :

project sbt version : 2.1

------------------------build.sbt------------------------------

name := "scala_sample"

version := "0.1"

scalaVersion := "2.11.12"

libraryDependencies ++= Seq(

  "org.apache.spark" %% "spark-core" % "2.1.0",

  "org.apache.spark" %% "spark-sql" % "2.1.0")

------------------------testing.scala------------------------------

import org.apache.spark.sql.{DataFrame,SparkSession}

import org.apache.spark.sql.functions._

object testing extends App{

  val spark = SparkSession.builder().master("local").appName("spark-shell").getOrCreate()

  import spark.implicits._

  val df1: DataFrame = Seq((1,1,"kjj"), (2,2,"987987")).toDF("col1", "col2","col3")

  val seq:Seq[String]=Seq("col1","col3")

  val obj:DF=new DF()

  obj.selectCols(df1,seq).show()

}

class DF{

  def selectCols(dataFrame: DataFrame,seq: Seq[String]): DataFrame ={

    val DFFiltered = dataFrame.select(seq.map(name => col(name)): _*)

    DFFiltered

  }

}

Result:

+----+------+

|col1|  col3|

+----+------+

|   1|   kjj|

|   2|987987|

+----+------+

Scala vs Python

Scala vs Python

----------------VAR / CONST ------------
val a:String="123" // Scala
var a:String="123" // Scala

a="123"# Python
--------------COMMENTS------------------
// this is a commented line in Scala
/* and this is a multiline comment,
...just choose! */

#Hi # Python
""" # Python
multi
"""
or
''' # Python
multi Line
'''
---------------PRINT-----------------
print("whatever you want") #  Python
println("whatever you want") // Scala

a="hi"
print(f"{a}")

val a="hi"
println(s"$a")

---------------LIST/ARRAY-----------------
Declaration :
var myArray = Array(2, 5, "apple", 78) // Scala
my_list = [2, 5, 'apple', 78] # Python

Indexing
myArray(1) // Scala 
my_list[1] # Python 

Slicing
myArray.slice(0,3) // Scala
my_list[0:3] # Python 


myArray.last    // Scala last element
myArray.max     // Scala maximum element
myArray.min     // Scala minimum element

my_list[-1]    # Python last element
max(my_list)   # Python maximum element
min(my_list)   # Python minimum element

sum(my_list) # Python- sum

myArray.sum    // scala sum
myArray.product  // multiplying elements in array

my_list.append('last words') # Python
myArray :+= "last words" 

---------------LOOP-----------------
for i in my_list:print(i)# Python
for (i <- myArray){ println(i)}// Scala
val list = List(1, 2, 3)
list.foreach(x => println(x)) 


--------------MAP------------------
list(map(lambda x:x*3,my_list)) # map in Python
myArray.map(i => i*3) // map in Scala

---------------FILTER-----------------
list(filter(lambda x:x%3==0,my_list)# filter in Python
myArray.filter(i => i%3 == 0) // filter in Scala

---------------REDUCE-----------------
from functools import reduce # Python
print(reduce(lambdax,y:x+y,(1,2,3))) #Python

val lst = List(1, 2, 3) // Scala
list.reduce((x, y) => x + y) // => 6 // Scala

---------------LIST COMPREHENSION-----------------
[i*3 for i in my_list if i%2 == 0] # list comprehension
myArray.filter(i => i%2 == 0).map(i => i*3) // Scala

---------------SORT-----------------------
sortBy(f: T => K): Seq[T]

---------------DICTIONARY-----------------
Declaration:
my_dict = {'first_name': 'Emma','age': 18}# Python
var myMap = (("firstName", "Emma"),("age", 18)) // Scala

New key:
my_dict['country_of_origin'] = 'Italy' # Python
myMap += ("countryOfOrigin" -> "Italy") // Scala

Access:
my_dict['first_name'] # Python
myMap("firstName") // Scala

Loop:
for key, value in my_dict.items(): # Python
    print(key)
    print(value)

for ((key, value) <- myMap){ // Scala
    println(key)
    println(value)
}
--------------TUPLE------------------
my_tup = (1, 2, 3) # Python
my_tup[0] # Python


myTup = (1, 2, 3) // Scala
myTup._1 // Scala
// the indexing is way different than arrays!

-------------SET-------------------
my_set = {1, 3, 5, 1} # Python
mySet = Set(1, 3, 5, 1) // Scala

-------------FUNCTION-------------------
def chop_string(input_string): # Python
    return input_string[0:5]

def chopString(inputString: String): String = { // Scala
    inputString.slice(0, 5)
}
--------------NAMED TUPLE/CASE CLASS------
from collections import namedtuple
n=namedtuple("n",["a","b"])
n2=n(10,20)
print(n2.a)

case class n(a:String,b:Int)
val n2: n = n("name", 1)
println(n2.a)
----------------SORTED 1---------------------
List(10, 5, 8, 1, 7).sortWith(_ < _) // Scala asc
sorted([10, 5, 8, 1, 7] , reverse =false) # Python

----------------SORTED 2---------------------
val list = List(('b',30),('c',10),('a',20))// Scala
println(list.sortBy(_._2))

l=[[1,2],[2,1]] # Python
sorted(l,key=lambdax:x[1])

----------------SORTED 3---------------------
List("banana", "pear", "apple", "orange").sortWith(_.length < _.length) // Scala
sorted(["banana", "pear", "apple", "orange"],key=lambda x:len(x)) # Python

output:(pear, apple, banana, orange)

-----------------Switch / Case ----------------
val result =1
result match {
case -1 =>{println("-1")}
case 0 => { println("asd")}
case x:Int if(x>0) =>
{ println(x)}
}

------------------OPTION = Some/getorelse/match---------------------
To eliminate NullPointerExceptions.

val age:Option[Integer] = None //Some(28)
val x=age.getOrElse(throw new Exception("error"))

scala : Perform basic test using MOCK

Scala : Perform basic test using MOCK and scalatest

Note :

1. If you already have scala installed in your system do not add in your scalaVersion := "2.13.0" build file.
2. Always add code in the test folder to use mock and scalatest.

build.sbt (I already have scala in my PC hence did not add scalaversion)

name := "sbt_test" version := "0.1"

//scalaVersion := "2.13.1"

libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.0" libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.4" % "test" libraryDependencies += "org.mockito" % "mockito-all" % "1.8.4" libraryDependencies += "org.scalamock" %% "scalamock" % "4.3.0" % "test" libraryDependencies += "org.testng" % "testng" % "6.10" 

 Under Project > src > test > scala > test1 > test_class.scala

package test1

import org.scalatest.{BeforeAndAfter, FunSpec, FunSuite}

import org.scalamock.scalatest.MockFactory

object test_class extends App with MockFactory{

 val x=mock[Point]

}

class Point(xc: Int, yc: Int) {

  var x: Int = xc

  var y: Int = yc

  def start(x:Int,y:Int): Unit ={

    println("start")

  }

  def move(dx: Int, dy: Int) {

    println("move")

  }

  def end(dx: Int) {

    println("end")

  }

}

Bazel : Debugging / JVM debugging using JDB for Scala - Bazel

Debugging / JVM debugging using JDB

Follow steps to perform debugging in Bazel / Any JVMs :

Steps :
1.cd path
2. Open new Terminal Window
3. bazel run build_name --java_debug
4. Open another Terminal Window
5. jdb -attach localhost:5005 -sourcepath ~/dev/remote/src/main/java/

Note :Breakpoint can only be set in Dev Code not on scalatest .

Eg:
#stop in Dev code
stop in dev_package.class_name.method

#stop at Dev Code
stop at dev_package.class_name:130

#Clear BreakPoint
clear dev_package.class_name:line_number
clear dev_package.class_name.method

Commands :
run
cont
step over
next
locals
print resultdf.show()


Ref : http://pages.cs.wisc.edu/~horwitz/jdb/jdb.html

scala : Basic Setup for Scala and Spark

Basic Setup for Scala

Scala can be run using :

1. Spark-Shell
2. SBT

Spark-Shell :Usually this is used as it provides scala along with spark capabilites to create RDD/DataSet and DF

Installation for Linux

1. java -version
2. sudo apt-get install scala #scala -version
1. type $scala
2. println("hi")
3. :q
3. goto - https://spark.apache.org/downloads.html
4. Download latest tar file
5. tar -zxvf spark-2.0.2-bin-hadoop2.7.tgz
6. cd spark-2.0.2-bin-hadoop2.7.tgz
1. ./spark-shell
2. :q //to quit
3. open .bashrc
7. add line SPARK_HOME=~/Downloads/spark-3.0.0-preview2-bin-hadoop2.7
8. $export PATH=$SPARK_HOME/bin:$PATH
9. $source ~/.bashrc
10. $spark-shell

Installation for mac

1. Run homebrew installer : $/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"#(visit homebrew for more details)
2. $xcode-select –install
3. $brew cask install java
4. $brew install scala
5. $brew install apache-spark
6. $spark-shell # to start spark-shell
7. $brew upgrade apache-spark #to Upgrade 

Start session 
$spark-shell

To Create a Dataframe:
:imports
#copy 4th point as 

#import org.apache.spark.sql.functions._ as  

#import org.apache.spark.sql._

import org.apache.spark.sql._

val mockDF1: DataFrame = Seq((0, "A"), (1, "B"), (0, "C")).toDF("col1", "col2")

mockDF1

TO uninstall 

brew uninstall scala

--------------

SBT shell

Linux

• Install brew
• open terminal
• Enter "/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
• After brew is installed successfully
• Enter"brew install sbt@1"
• TO uninstall - brew uninstall scala

Linux :

Ref : http://www.codebind.com/linux-tutorials/install-scala-sbt-java-ubuntu-18-04-lts-linux/

https://www.scala-sbt.org/1.x/docs/Installing-sbt-on-Linux.html

Getting SBT Started in Terminal:

1. Open Terminal
2. enter "sbt"
3. After sbt shell is opened
4. Open Scala REPL session inside SBT using - "console" or "consoleQuick"
5. Type "println("helloworld")
6. To quit type ":q" or":quit"
7. And "exit" to exit sbt shell

Note : You can only do basic operations here . But cannot do operations where there is dependency on libraries .For that you need a build tool like sbt or bazel .

To add library into your sbt shell :

1. Download required jar file Eg:https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.12/2.4.4
2. Open sbt shell
3. run cmd ":require spark-sql_2.12.jar" #Added - file.jar

import org.apache.spark.sql.{Column, DataFrame, SaveMode, SparkSession}

Using SBT in IntelliJ :

1. IntelliJ by default comes with sbt
2. Install Idea intelliJ
3. Create a new sbt project
4. goto project /src/main/test
5. Create a new file "test1.scala"
6. copy below code  
7. set SBT SDT to 2.11
8. add contents into build file
9. Run it (rt ck run)

build.sbt:

name := "sbt_test" version := "0.1" scalaVersion := "2.11.8"
libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % "2.1.0", "org.apache.spark" %% "spark-sql" % "2.1.0")

libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.0"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.4" % "test"
libraryDependencies += "org.mockito" % "mockito-all" % "1.8.4"
libraryDependencies += "org.scalamock" %% "scalamock" % "4.3.0" % "test"
libraryDependencies += "org.testng" % "testng" % "6.10"

//Basic Class:

object test1 extends App{

  println("Hello World")

}

or 

object test1 { def

            main(args:

              Array[String]): Unit = println("Hello, World!") }

or

import org.apache.spark.sql.SparkSession

import org.scalamock.scalatest.MockFactory

import org.apache.spark.sql.{Column, DataFrame, SaveMode, SparkSession}

import org.apache.spark.sql.functions._

import org.apache.spark.sql.types._

object test1  extends App with MockFactory {

  val spark = SparkSession.builder().master("local").appName("spark-shell").getOrCreate()

  import spark.implicits._

  val df1: DataFrame = Seq((1,1), (2,2), (3,3)).toDF("col1", "col2")

  df1.show()

}

Issues :

1. Error:scalac: Multiple 'scala-library*.jar' files (scala-library.jar, scala-library.jar, scala-library.jar) in

          Scala

            compiler classpath in

          Scala

            SDK scala-sdk-2.11.7`

Solution:

File > Project_Structure > Libraries     

Remove "SBT:org.scala-lang:scala-library:2.11.8:jar"

2. Cannot resolve App

Solution:

Set the current Scala SDT to 2.11.8

--------------------------------------------------------------------------------------------------------

Issues :

1. Error:scalac: Multiple 'scala-library*.jar' files (scala-library.jar, scala-library.jar, scala-library.jar) in

        Scala

          compiler classpath in

        Scala

          SDK scala-sdk-2.11.7`

Solution : Goto File > Project Stucture > Platform Settings > SDKs > Remove Duplicate
Solution 2: Remove "scalaVersion := "2.xx.xx" from build.sbt file
Solution 3:

2. Could not find or load main class in scala in intellij IDE
Solution: Right click on "src folder" and select Mark Directory as -> Sources root

--------------------------------------------------------------------------------------------------------

Ref:

https://www.analyticsvidhya.com/blog/2017/01/scala/

ScalaTest : testng

ScalaTest : TestNG

Pre-Req :

• Os - Linux ubuntu / debian
• Java /jdk 1.8 +
• scala 2.11 +
• Intellij
• Install sbt plugin in Intelli
• Create an scala project in Intelli using sbt

Commands:
scala -version
sbt sbtVersion

----build.sbt (add below line)--------
scalaVersion := "x.xx.0"
libraryDependencies += "org.scalatest" % "scalatest_x.xx" % "3.0.8" % "test"
libraryDependencies += "org.testng" % "testng" % "6.10"

--- install sbt----
open terminal
echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823
sudo apt-get update
sudo apt-get install sbt
---------------------

Location : src/test/scala/ExampleTestNGSuite.scala

//https://searchcode.com/codesearch/view/16471591/
//http://doc.scalatest.org/1.8/index.html#package

import org.scalatestplus.testng.TestNGSuite
import org.testng.annotations.Test
import org.testng.annotations.BeforeMethod
import org.testng.annotations.BeforeClass
import org.testng.annotations.BeforeSuite
import org.testng.annotations.AfterMethod
import org.testng.annotations.AfterClass
import org.testng.annotations.AfterSuite
import org.testng.annotations.DataProvider

class ExampleTestNGSuite extends TestNGSuite {

// @AfterSuite def failAfterSuite(){ throw new Exception("fail in before method") }
@BeforeMethod def passBeforeMethod(){println("b3")}
@BeforeClass def passBeforeClass(){println("b2")}
@BeforeSuite def passBeforeSuite(){println("b1")}
@AfterMethod def passAfterMethod(){println("a2")}
@AfterClass def passAfterClass(){println("a3")}
@AfterSuite def passAfterSuite(){println("a1")}
@Test(invocationCount = 10)
def thisTestRunsTenTimes = {}
/*
@Test(groups = Array("runMe"))
def testWithException(){
throw new Exception("exception!!!")
}
@Test(groups = Array("runMe"))
def testWithAssertFail = assert( 1 === 2, "assert fail!!!" )

@Test(dependsOnMethods = Array("testWithException"))
def testToGetSkipped = {}
*/
@DataProvider(name = "any_name")
def andValues = {
val test_var = Array("0", "1")
for( x <- test_var; y <- test_var ) yield Array(x,y)
}

@Test(dataProvider = "any_name")
def testAndStates(a: String, b: String){
println("a="+ a + ",b="+ b)
}
}

-----Run Sbt in terminal -----
cd to projects >
run command "sbt test"

------------------------

Result :
A testNG report has to be generated in the project folder .

Testing with Scala / SBT

Testing with Scala / SBT

Pre-Req :
Os - Linux ubuntu / debian
Java /jdk 1.8 +
scala 2.11 +
Intellij
Install sbt plugin in Intelli
Create an scala project in Intelli using sbt

Commands:
scala -version
sbt sbtVersion

----build.sbt (add below line)--------
scalaVersion := "2.xx.0"
libraryDependencies += "org.scalatest" % "scalatest_2.xx" % "3.0.8" % "test"
libraryDependencies +="org.pegdown" % "pegdown" % "1.4.2" % "test"
libraryDependencies += "org.mockito" % "mockito-all" % "1.8.4"
libraryDependencies += "org.scalamock" %% "scalamock" % "4.3.0" % "test"
testOptions in Test += Tests.Argument(TestFrameworks.ScalaTest, "-h", "target/test-reports")
(click on auto-import link when it appears)


--- install sbt----
open terminal
echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823
sudo apt-get update
sudo apt-get install sbt

---- Config sbt in IntelliJ----
to Debug/Run test in sbt:
1. Edit Configurations
2. "+" Add Scala Test
3. In the new file , Select the test file (test.scala)
4. Check "use sbt"
5. Apply and OK
6. rt ck on test file in File structure/ editor / tool bar - ck run/Debug
-----Run Sbt in terminal -----
cd to  projects >
run command "sbt test"
---------------------

Src File : src/main/scala/sqCalculator.scala
object sqCalculator extends App {
  def sq(x: Int) = {
    x * x
  }
}

Test File : test/scala/sqCalculatorTest.scala
import org.scalatest.FunSuite
class sqCalculatorTest extends FunSuite {
  test("sqCalculator.cube") {
    if(sqCalculator.sq(3) === 9){
      println ("passed")
      assert (true)
    }
    else{println (false)}
  }
}
---------------------//Mockito----------------------------
//test/scala/LoginService.scala 

trait LoginService {
  def login(name: String, password: String): String
}

//test/scala/LoginServiceTest.scala
import org.scalatest.FunSuite
import org.scalatest.BeforeAndAfter
import org.scalatestplus.mockito.MockitoSugar
import org.mockito.Mockito._

class LoginServiceTest extends FunSuite with BeforeAndAfter with MockitoSugar {
  before {
    println("Start")
  }
  after {
  println("Ends")
  }

  test ("test login service"){
    val service = mock[LoginService]
    when(service.login("johndoe", "secret")).thenReturn(("johndoe"))
    when(service.login("joehacker", "secret")).thenReturn("an")
    val johndoe = service.login("johndoe", "secret")
    val joehacker = service.login("joehacker", "secret")
    if(johndoe == "johndoe"){
      assert(true)
    }
    if(joehacker == None)
      assert(false)
    else
        assert(true)
  }
}

----------------ScalaMock--------------------------------
//test/scala/LoginServiceTest2.scala
import org.scalatest.{BeforeAndAfter, FunSpec, FunSuite}
import org.scalamock.scalatest.MockFactory

class LoginServiceTest2 extends FunSpec with BeforeAndAfter with MockFactory  {
  before{
    println("LoginServiceTest2 - start")
  }
  it("val test") {
  val any_name: LoginService = mock[LoginService]
    (any_name.login _) expects("a","a") returning "hi"
    println(any_name.login("a","a"))
  }
  after{
    println("LoginServiceTest2 - end")
  }
}