Git : Setting up a clone project

Git Pre- Req

Computer Authentication :

Pre- Req:

Windows:

                         i.         Open https://git-scm.com/download/win

                       ii.         Install (Next > next >…)

                      iii.         Right click on select git bash option

Mac :

·       Open Terminal

Steps :

1.     Enter ssh-keygen -t rsa in your git bash prompt / terminal (type “yes” when asked and press enter key otherwise)

2.     Note down location of the file created or usually file gets created in below locations

3.     Open id_rsa.pub file

a.     For Windows :   C:\User_name\.ssh\id_rsa.pub or  C:\xxx_USER \.ssh\id_rsa.pub

b.     For Mac :          /Users/User_name/.ssh/id_rsa.pub

4.     Copy entire Contents

Eg :

ssh-rsa sadfasfldfllflamflamsflWWQEW;as

--------

--------

---------------------------------@in.xxx.com

5.    Create a new git hub account

6.     Goto Settings

7.     Click on SSH and GPG keys

8.     Click on New SSH Keys

9.     Paste the contents inside

10.  Click on Add button

11.  Create a new folder in your system with name git_test (any location)

12.  Navigate inside the folder

13.   

a.     For windows : Open git bash by right click

b.     For mac : use terminal to navigate inside the new folder using “cd command”

14.  In your terminal / git bash run command (copy link below from the git project online)

“ git clone git@github.xxx.com:djayapr1/testing.git”

15.  Run command “cd” into the cloned folder

scala : Basic Setup for Scala and Spark

Basic Setup for Scala

Scala can be run using :

1. Spark-Shell
2. SBT

Spark-Shell :Usually this is used as it provides scala along with spark capabilites to create RDD/DataSet and DF

Installation for Linux

1. java -version
2. sudo apt-get install scala #scala -version
1. type $scala
2. println("hi")
3. :q
3. goto - https://spark.apache.org/downloads.html
4. Download latest tar file
5. tar -zxvf spark-2.0.2-bin-hadoop2.7.tgz
6. cd spark-2.0.2-bin-hadoop2.7.tgz
1. ./spark-shell
2. :q //to quit
3. open .bashrc
7. add line SPARK_HOME=~/Downloads/spark-3.0.0-preview2-bin-hadoop2.7
8. $export PATH=$SPARK_HOME/bin:$PATH
9. $source ~/.bashrc
10. $spark-shell

Installation for mac

1. Run homebrew installer : $/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"#(visit homebrew for more details)
2. $xcode-select –install
3. $brew cask install java
4. $brew install scala
5. $brew install apache-spark
6. $spark-shell # to start spark-shell
7. $brew upgrade apache-spark #to Upgrade 

Start session 
$spark-shell

To Create a Dataframe:
:imports
#copy 4th point as 

#import org.apache.spark.sql.functions._ as  

#import org.apache.spark.sql._

import org.apache.spark.sql._

val mockDF1: DataFrame = Seq((0, "A"), (1, "B"), (0, "C")).toDF("col1", "col2")

mockDF1

TO uninstall 

brew uninstall scala

--------------

SBT shell

Linux

• Install brew
• open terminal
• Enter "/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
• After brew is installed successfully
• Enter"brew install sbt@1"
• TO uninstall - brew uninstall scala

Linux :

Ref : http://www.codebind.com/linux-tutorials/install-scala-sbt-java-ubuntu-18-04-lts-linux/

https://www.scala-sbt.org/1.x/docs/Installing-sbt-on-Linux.html

Getting SBT Started in Terminal:

1. Open Terminal
2. enter "sbt"
3. After sbt shell is opened
4. Open Scala REPL session inside SBT using - "console" or "consoleQuick"
5. Type "println("helloworld")
6. To quit type ":q" or":quit"
7. And "exit" to exit sbt shell

Note : You can only do basic operations here . But cannot do operations where there is dependency on libraries .For that you need a build tool like sbt or bazel .

To add library into your sbt shell :

1. Download required jar file Eg:https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.12/2.4.4
2. Open sbt shell
3. run cmd ":require spark-sql_2.12.jar" #Added - file.jar

import org.apache.spark.sql.{Column, DataFrame, SaveMode, SparkSession}

Using SBT in IntelliJ :

1. IntelliJ by default comes with sbt
2. Install Idea intelliJ
3. Create a new sbt project
4. goto project /src/main/test
5. Create a new file "test1.scala"
6. copy below code  
7. set SBT SDT to 2.11
8. add contents into build file
9. Run it (rt ck run)

build.sbt:

name := "sbt_test" version := "0.1" scalaVersion := "2.11.8"
libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % "2.1.0", "org.apache.spark" %% "spark-sql" % "2.1.0")

libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.0"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.4" % "test"
libraryDependencies += "org.mockito" % "mockito-all" % "1.8.4"
libraryDependencies += "org.scalamock" %% "scalamock" % "4.3.0" % "test"
libraryDependencies += "org.testng" % "testng" % "6.10"

//Basic Class:

object test1 extends App{

  println("Hello World")

}

or 

object test1 { def

            main(args:

              Array[String]): Unit = println("Hello, World!") }

or

import org.apache.spark.sql.SparkSession

import org.scalamock.scalatest.MockFactory

import org.apache.spark.sql.{Column, DataFrame, SaveMode, SparkSession}

import org.apache.spark.sql.functions._

import org.apache.spark.sql.types._

object test1  extends App with MockFactory {

  val spark = SparkSession.builder().master("local").appName("spark-shell").getOrCreate()

  import spark.implicits._

  val df1: DataFrame = Seq((1,1), (2,2), (3,3)).toDF("col1", "col2")

  df1.show()

}

Issues :

1. Error:scalac: Multiple 'scala-library*.jar' files (scala-library.jar, scala-library.jar, scala-library.jar) in

          Scala

            compiler classpath in

          Scala

            SDK scala-sdk-2.11.7`

Solution:

File > Project_Structure > Libraries     

Remove "SBT:org.scala-lang:scala-library:2.11.8:jar"

2. Cannot resolve App

Solution:

Set the current Scala SDT to 2.11.8

--------------------------------------------------------------------------------------------------------

Issues :

1. Error:scalac: Multiple 'scala-library*.jar' files (scala-library.jar, scala-library.jar, scala-library.jar) in

        Scala

          compiler classpath in

        Scala

          SDK scala-sdk-2.11.7`

Solution : Goto File > Project Stucture > Platform Settings > SDKs > Remove Duplicate
Solution 2: Remove "scalaVersion := "2.xx.xx" from build.sbt file
Solution 3:

2. Could not find or load main class in scala in intellij IDE
Solution: Right click on "src folder" and select Mark Directory as -> Sources root

--------------------------------------------------------------------------------------------------------

Ref:

https://www.analyticsvidhya.com/blog/2017/01/scala/

XML to HTML report using xsl

XML to HTML report using xsl

Sample junit xml report

<testsuite errors="0" failures="2" hostname="Deepaks-MacBook-Air.local" name="xxx.scalaTests.common.DefaultColumnAdditionSpec" tests="4" time="7.66" timestamp="2019-08-10T09:58:56">

<testcase name="1To validate resultant df, when input datatype is of type string. 1To validate resultant df, when input datatype is of type string." classname="xxx.scalaTests.common.DefaultColumnAdditionSpec" time="7.625"> </testcase>

<testcase name="3To validate resultant df, when input datatype is of type string. 3To validate resultant df, when input datatype is of type string." classname="xxx.scalaTests.common.DefaultColumnAdditionSpec" time="0.003"> </testcase>

<testcase name="4To validate resultant df, when input datatype is of type string. 4To validate resultant df, when input datatype is of type string." classname="xxx.scalaTests.common.DefaultColumnAdditionSpec" time="0.008">
<failure message="org.scalatest.exceptions.TestFailedException was thrown." type="class org.scalatest.exceptions.TestFailedException">...</failure>
</testcase>

<testcase name="5To validate resultant df, when input datatype is of type string. 5To validate resultant df, when input datatype is of type string." classname="xxx.scalaTests.common.DefaultColumnAdditionSpec" time="0.0">
<failure message="org.scalatest.exceptions.TestFailedException was thrown." type="class org.scalatest.exceptions.TestFailedException">...</failure>
</testcase>

</testsuite>

----save as test.xsl--------
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="/">
<html> 
<body>
  <h2>Result</h2>
  <table border="1">
    <tr bgcolor="#9acd32">
      <th style="text-align:left">Test Name</th>
      <th style="text-align:left">Result</th>
    </tr>
    <xsl:for-each select="testsuites /testsuite/testcase">
    <tr>
      <td><xsl:value-of select="@name"/></td>
<xsl:choose>
<xsl:when test="failure"><td>FAIL</td></xsl:when>
<xsl:otherwise><td>PASS</td></xsl:otherwise>
</xsl:choose>
    </tr>
    </xsl:for-each>
  </table>
</body>
</html>
</xsl:template>
</xsl:stylesheet>
----------------

STEPS:

1. Save the above xsl code as test.xsl
2. Open terminal
3. cd to juit report file.
4. Run below code 
1. MAC : xsltproc -o result.html test.xsl junit_result.xml 
2. Win :  msxsl sourcefile.xml test.xsl -o junit_result.html 

ScalaTest : testng

ScalaTest : TestNG

Pre-Req :

• Os - Linux ubuntu / debian
• Java /jdk 1.8 +
• scala 2.11 +
• Intellij
• Install sbt plugin in Intelli
• Create an scala project in Intelli using sbt

Commands:
scala -version
sbt sbtVersion

----build.sbt (add below line)--------
scalaVersion := "x.xx.0"
libraryDependencies += "org.scalatest" % "scalatest_x.xx" % "3.0.8" % "test"
libraryDependencies += "org.testng" % "testng" % "6.10"

--- install sbt----
open terminal
echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823
sudo apt-get update
sudo apt-get install sbt
---------------------

Location : src/test/scala/ExampleTestNGSuite.scala

//https://searchcode.com/codesearch/view/16471591/
//http://doc.scalatest.org/1.8/index.html#package

import org.scalatestplus.testng.TestNGSuite
import org.testng.annotations.Test
import org.testng.annotations.BeforeMethod
import org.testng.annotations.BeforeClass
import org.testng.annotations.BeforeSuite
import org.testng.annotations.AfterMethod
import org.testng.annotations.AfterClass
import org.testng.annotations.AfterSuite
import org.testng.annotations.DataProvider

class ExampleTestNGSuite extends TestNGSuite {

// @AfterSuite def failAfterSuite(){ throw new Exception("fail in before method") }
@BeforeMethod def passBeforeMethod(){println("b3")}
@BeforeClass def passBeforeClass(){println("b2")}
@BeforeSuite def passBeforeSuite(){println("b1")}
@AfterMethod def passAfterMethod(){println("a2")}
@AfterClass def passAfterClass(){println("a3")}
@AfterSuite def passAfterSuite(){println("a1")}
@Test(invocationCount = 10)
def thisTestRunsTenTimes = {}
/*
@Test(groups = Array("runMe"))
def testWithException(){
throw new Exception("exception!!!")
}
@Test(groups = Array("runMe"))
def testWithAssertFail = assert( 1 === 2, "assert fail!!!" )

@Test(dependsOnMethods = Array("testWithException"))
def testToGetSkipped = {}
*/
@DataProvider(name = "any_name")
def andValues = {
val test_var = Array("0", "1")
for( x <- test_var; y <- test_var ) yield Array(x,y)
}

@Test(dataProvider = "any_name")
def testAndStates(a: String, b: String){
println("a="+ a + ",b="+ b)
}
}

-----Run Sbt in terminal -----
cd to projects >
run command "sbt test"

------------------------

Result :
A testNG report has to be generated in the project folder .

Testing with Scala / SBT

Testing with Scala / SBT

Pre-Req :
Os - Linux ubuntu / debian
Java /jdk 1.8 +
scala 2.11 +
Intellij
Install sbt plugin in Intelli
Create an scala project in Intelli using sbt

Commands:
scala -version
sbt sbtVersion

----build.sbt (add below line)--------
scalaVersion := "2.xx.0"
libraryDependencies += "org.scalatest" % "scalatest_2.xx" % "3.0.8" % "test"
libraryDependencies +="org.pegdown" % "pegdown" % "1.4.2" % "test"
libraryDependencies += "org.mockito" % "mockito-all" % "1.8.4"
libraryDependencies += "org.scalamock" %% "scalamock" % "4.3.0" % "test"
testOptions in Test += Tests.Argument(TestFrameworks.ScalaTest, "-h", "target/test-reports")
(click on auto-import link when it appears)


--- install sbt----
open terminal
echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823
sudo apt-get update
sudo apt-get install sbt

---- Config sbt in IntelliJ----
to Debug/Run test in sbt:
1. Edit Configurations
2. "+" Add Scala Test
3. In the new file , Select the test file (test.scala)
4. Check "use sbt"
5. Apply and OK
6. rt ck on test file in File structure/ editor / tool bar - ck run/Debug
-----Run Sbt in terminal -----
cd to  projects >
run command "sbt test"
---------------------

Src File : src/main/scala/sqCalculator.scala
object sqCalculator extends App {
  def sq(x: Int) = {
    x * x
  }
}

Test File : test/scala/sqCalculatorTest.scala
import org.scalatest.FunSuite
class sqCalculatorTest extends FunSuite {
  test("sqCalculator.cube") {
    if(sqCalculator.sq(3) === 9){
      println ("passed")
      assert (true)
    }
    else{println (false)}
  }
}
---------------------//Mockito----------------------------
//test/scala/LoginService.scala 

trait LoginService {
  def login(name: String, password: String): String
}

//test/scala/LoginServiceTest.scala
import org.scalatest.FunSuite
import org.scalatest.BeforeAndAfter
import org.scalatestplus.mockito.MockitoSugar
import org.mockito.Mockito._

class LoginServiceTest extends FunSuite with BeforeAndAfter with MockitoSugar {
  before {
    println("Start")
  }
  after {
  println("Ends")
  }

  test ("test login service"){
    val service = mock[LoginService]
    when(service.login("johndoe", "secret")).thenReturn(("johndoe"))
    when(service.login("joehacker", "secret")).thenReturn("an")
    val johndoe = service.login("johndoe", "secret")
    val joehacker = service.login("joehacker", "secret")
    if(johndoe == "johndoe"){
      assert(true)
    }
    if(joehacker == None)
      assert(false)
    else
        assert(true)
  }
}

----------------ScalaMock--------------------------------
//test/scala/LoginServiceTest2.scala
import org.scalatest.{BeforeAndAfter, FunSpec, FunSuite}
import org.scalamock.scalatest.MockFactory

class LoginServiceTest2 extends FunSpec with BeforeAndAfter with MockFactory  {
  before{
    println("LoginServiceTest2 - start")
  }
  it("val test") {
  val any_name: LoginService = mock[LoginService]
    (any_name.login _) expects("a","a") returning "hi"
    println(any_name.login("a","a"))
  }
  after{
    println("LoginServiceTest2 - end")
  }
}

Python : Pytest-selenium without setup/installation of driver

Python : Pytest-selenium without setup/installation of driver

project/requirements.txt

pandas >= 0.24.2
numpy >= 1.16.3
pytest >= 3.10.1
pytest-html >= 1.20.0
pytest-metadata >= 1.8.0
pytest-profiling >= 1.6.0
pyderman >= 1.3.0
pytest-selenium >= 1.16.0

project/tests/test_case1.py

import pytest
from lib import csv_reader
import os
import pyderman as dr
from pathlib import Path



link="https://www.stats.govt.nz/assets/Uploads/Electronic-card-transactions/Electronic-card-transactions-May-2019/Download-data/electronic-card-transaction-may-2019-csv.zip"
import pytest
@pytest.fixturedef firefox_options(firefox_options):
    #firefox_options.binary = '/path/to/firefox-bin'    

       destination = str(Path(__file__).resolve().parents[1]) + "/temp/"    destination="/home/deepak/PycharmProjects/csv_test/temp"    

   if(os.path.exists(destination)):
        print(destination)
        print("paht exists")
    destination=destination.replace("\\","\\\\")
    firefox_options.add_argument('-foreground')
    firefox_options.set_preference('browser.download.folderList', 2)  # custom location    firefox_options.set_preference('browser.download.manager.showWhenStarting', False)
    firefox_options.set_preference('browser.download.dir', destination)
    firefox_options.set_preference("browser.helperApps.neverAsk.saveToDisk", "application/zip")
    return firefox_options

def test_deep():
    if(1==1):
        path = dr.install(browser=dr.firefox)
        print('Installed geckodriver driver to path: %s' % path)
        print("yes")
        assert True    else:
        print("no")
        assert False

def test_example(selenium):
    selenium.get(link)

--------------------------

To RUn :

1. open terminal

2. cd inside project

3. pip install requirements.txt

4. pytest

Python : Pytest logger

 Python : Pytest logger

Create file : /project/pytest.ini

[pytest]

testpaths = tests/

python_paths = ./

console_output_style = progress

addopts = -rsxX -q

log_cli = 1

log_cli_level = INFO

log_cli_format = %(asctime)s [%(levelname)8s] %(message)s (%(filename)s:%(lineno)s)

log_cli_date_format=%Y-%m-%d %H:%M:%S

log_file_date_format = %Y-%m-%d %H:%M:%S

log_file_format = %(asctime)s %(levelname)s %(message)s

log_date_format = %Y-%m-%d %H:%M:%S

log_file=simple_logger.log

log_print = True

python_files =

    test_*.py

    *_test.py

 

 

Using without class :
LOGGER = logging.getLogger(__name__) #Decalared as a global
LOGGER.error("Unable to make connection to DB")#to log fails inside/ouside def

Usage in a class :
self.LOGGER = logging.getLogger(__name__)#inside a constructor
self.LOGGER.error("Unable to make connection to DB")#to log fails

Python : Config /Properties/ ini file

Python : Config/ini/Properties file

-----------credentials.config-----------

[db]
dbname= dbname
hostname= host_anme
port= 1234
username= usr_name
password= pass

----------------------

Python : Config /Properties/ ini file

from configparser import RawConfigParser

config_path = os.getcwd()+'/properties/credentials.config'

confparser = RawConfigParser()

with open(config_path, "r") as config_file:

    confparser.read_file(config_file)

KWARGS = {

    "dbname": confparser["db"]["dbname"],

    "hostname": confparser["db"]["hostname"],

    "port":confparser["db"]["port"],

    "username": confparser["db"]["username"],

    "password": confparser["db"]["password"]

        }

"""

MongoDb : MongoDb no SQL