Testing with Scala / SBT

Testing with Scala / SBT

Pre-Req :
Os - Linux ubuntu / debian
Java /jdk 1.8 +
scala 2.11 +
Intellij
Install sbt plugin in Intelli
Create an scala project in Intelli using sbt

Commands:
scala -version
sbt sbtVersion

----build.sbt (add below line)--------
scalaVersion := "2.xx.0"
libraryDependencies += "org.scalatest" % "scalatest_2.xx" % "3.0.8" % "test"
libraryDependencies +="org.pegdown" % "pegdown" % "1.4.2" % "test"
libraryDependencies += "org.mockito" % "mockito-all" % "1.8.4"
libraryDependencies += "org.scalamock" %% "scalamock" % "4.3.0" % "test"
testOptions in Test += Tests.Argument(TestFrameworks.ScalaTest, "-h", "target/test-reports")
(click on auto-import link when it appears)


--- install sbt----
open terminal
echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823
sudo apt-get update
sudo apt-get install sbt

---- Config sbt in IntelliJ----
to Debug/Run test in sbt:
1. Edit Configurations
2. "+" Add Scala Test
3. In the new file , Select the test file (test.scala)
4. Check "use sbt"
5. Apply and OK
6. rt ck on test file in File structure/ editor / tool bar - ck run/Debug
-----Run Sbt in terminal -----
cd to  projects >
run command "sbt test"
---------------------

Src File : src/main/scala/sqCalculator.scala
object sqCalculator extends App {
  def sq(x: Int) = {
    x * x
  }
}

Test File : test/scala/sqCalculatorTest.scala
import org.scalatest.FunSuite
class sqCalculatorTest extends FunSuite {
  test("sqCalculator.cube") {
    if(sqCalculator.sq(3) === 9){
      println ("passed")
      assert (true)
    }
    else{println (false)}
  }
}
---------------------//Mockito----------------------------
//test/scala/LoginService.scala 

trait LoginService {
  def login(name: String, password: String): String
}

//test/scala/LoginServiceTest.scala
import org.scalatest.FunSuite
import org.scalatest.BeforeAndAfter
import org.scalatestplus.mockito.MockitoSugar
import org.mockito.Mockito._

class LoginServiceTest extends FunSuite with BeforeAndAfter with MockitoSugar {
  before {
    println("Start")
  }
  after {
  println("Ends")
  }

  test ("test login service"){
    val service = mock[LoginService]
    when(service.login("johndoe", "secret")).thenReturn(("johndoe"))
    when(service.login("joehacker", "secret")).thenReturn("an")
    val johndoe = service.login("johndoe", "secret")
    val joehacker = service.login("joehacker", "secret")
    if(johndoe == "johndoe"){
      assert(true)
    }
    if(joehacker == None)
      assert(false)
    else
        assert(true)
  }
}

----------------ScalaMock--------------------------------
//test/scala/LoginServiceTest2.scala
import org.scalatest.{BeforeAndAfter, FunSpec, FunSuite}
import org.scalamock.scalatest.MockFactory

class LoginServiceTest2 extends FunSpec with BeforeAndAfter with MockFactory  {
  before{
    println("LoginServiceTest2 - start")
  }
  it("val test") {
  val any_name: LoginService = mock[LoginService]
    (any_name.login _) expects("a","a") returning "hi"
    println(any_name.login("a","a"))
  }
  after{
    println("LoginServiceTest2 - end")
  }
}

Python : Pytest-selenium without setup/installation of driver

Python : Pytest-selenium without setup/installation of driver

project/requirements.txt

pandas >= 0.24.2
numpy >= 1.16.3
pytest >= 3.10.1
pytest-html >= 1.20.0
pytest-metadata >= 1.8.0
pytest-profiling >= 1.6.0
pyderman >= 1.3.0
pytest-selenium >= 1.16.0

project/tests/test_case1.py

import pytest
from lib import csv_reader
import os
import pyderman as dr
from pathlib import Path



link="https://www.stats.govt.nz/assets/Uploads/Electronic-card-transactions/Electronic-card-transactions-May-2019/Download-data/electronic-card-transaction-may-2019-csv.zip"
import pytest
@pytest.fixturedef firefox_options(firefox_options):
    #firefox_options.binary = '/path/to/firefox-bin'    

       destination = str(Path(__file__).resolve().parents[1]) + "/temp/"    destination="/home/deepak/PycharmProjects/csv_test/temp"    

   if(os.path.exists(destination)):
        print(destination)
        print("paht exists")
    destination=destination.replace("\\","\\\\")
    firefox_options.add_argument('-foreground')
    firefox_options.set_preference('browser.download.folderList', 2)  # custom location    firefox_options.set_preference('browser.download.manager.showWhenStarting', False)
    firefox_options.set_preference('browser.download.dir', destination)
    firefox_options.set_preference("browser.helperApps.neverAsk.saveToDisk", "application/zip")
    return firefox_options

def test_deep():
    if(1==1):
        path = dr.install(browser=dr.firefox)
        print('Installed geckodriver driver to path: %s' % path)
        print("yes")
        assert True    else:
        print("no")
        assert False

def test_example(selenium):
    selenium.get(link)

--------------------------

To RUn :

1. open terminal

2. cd inside project

3. pip install requirements.txt

4. pytest

Python : Pytest logger

 Python : Pytest logger

Create file : /project/pytest.ini

[pytest]

testpaths = tests/

python_paths = ./

console_output_style = progress

addopts = -rsxX -q

log_cli = 1

log_cli_level = INFO

log_cli_format = %(asctime)s [%(levelname)8s] %(message)s (%(filename)s:%(lineno)s)

log_cli_date_format=%Y-%m-%d %H:%M:%S

log_file_date_format = %Y-%m-%d %H:%M:%S

log_file_format = %(asctime)s %(levelname)s %(message)s

log_date_format = %Y-%m-%d %H:%M:%S

log_file=simple_logger.log

log_print = True

python_files =

    test_*.py

    *_test.py

 

 

Using without class :
LOGGER = logging.getLogger(__name__) #Decalared as a global
LOGGER.error("Unable to make connection to DB")#to log fails inside/ouside def

Usage in a class :
self.LOGGER = logging.getLogger(__name__)#inside a constructor
self.LOGGER.error("Unable to make connection to DB")#to log fails

Python : Config /Properties/ ini file

Python : Config/ini/Properties file

-----------credentials.config-----------

[db]
dbname= dbname
hostname= host_anme
port= 1234
username= usr_name
password= pass

----------------------

Python : Config /Properties/ ini file

from configparser import RawConfigParser

config_path = os.getcwd()+'/properties/credentials.config'

confparser = RawConfigParser()

with open(config_path, "r") as config_file:

    confparser.read_file(config_file)

KWARGS = {

    "dbname": confparser["db"]["dbname"],

    "hostname": confparser["db"]["hostname"],

    "port":confparser["db"]["port"],

    "username": confparser["db"]["username"],

    "password": confparser["db"]["password"]

        }

"""

MongoDb : MongoDb no SQL

pandas : Lambda ,Map and Eval Conditions

Lambda ,Map and eval in Pandas

1. Simple 

import pandas as pd

a=pd.DataFrame({"a":["deea","123","deep","deepak"]})

b=pd.DataFrame({"b":["deeap","1234","deepak","deepa"]})

c=pd.DataFrame(map(lambda x,y:x in y,a['a'],b['b']))

print(c)

 

 

result : 

       0

0   True

1   True

2   True

3  False 

 

2. Using Dynamic Formula :

import pandas as pd

a=pd.DataFrame({"a":["deea","123","deep","deepak"]})

b=pd.DataFrame({"b":["deeap","1234","deepak","deepa"]})

c=pd.DataFrame(map(eval(input("enter -lambda x,y:x in y")),a['a'].values,b['b'].values))

print(c)

       0

0   True

1   True

2   True

3  False

 

3)Using Dynamic Formula and 1st columns

import pandas as pd

a=pd.DataFrame({"a":["deea","123","deep","deepak"]})

b=pd.DataFrame({"b":["deeap","1234","deepak","deepa"]})

c=pd.DataFrame(map(eval(input("enter -lambda x,y:x in y")),a.iloc[:,0],b.iloc[:,0]))

print(c)

  

Result: 

  0

0 True

1 True

2 True

3 False 

 

"""

"""

Column Names

data.iloc[:,0] # first column of data frame (first_name)
data.iloc[:,1] # second column of data frame (last_name)
data.iloc[:,-1] # last column of data frame (id)

Rows and Columns
data.iloc[0:5] # first five rows of dataframe
data.iloc[:, 0:2] # first two columns of data frame with all rows
data.iloc[[0,3,6,24], [0,5,6]] # 1st, 4th, 7th, 25th row + 1st 6th 7th columns.
data.iloc[0:5, 5:8] # first 5 rows and 5th, 6th, 7th columns of data frame
"""

Numpy : Lambda / map / vectorize

Numpy : Lambda / map / vectorize

import numpy as np

def myfunc(a, b):

    if a > b:

        return a - b

    else:

        return a + b

n1=np.array([1,2,3])

n2=np.array([5,4,3])

vfunc = np.vectorize(myfunc)

print(vfunc(n1,n2))

Output :

[6 6 6]

pandas : Import snippet

pandas : Import snippet

TC_ID	TC_NAME	QUERY1	QUERY2	CONDITION	BASICTEST(QUERY1/QUERY2)
TC0001	TC_NAME1	Select col1 from table1	Select col1 from table2	1	11
TC0002	TC_NAME2	Select col1 from table2	Select col1 from table3	2	12
TC0003	TC_NAME3	Select col1 from table3	Select col1 from table4	3	13
TC0004	TC_NAME4	Select col1 from table4	Select col1 from table5	4	14
TC0005	TC_NAME5	Select col1 from table5	Select col1 from table6	5	15
TC0006	TC_NAME6	Select col1 from table6	Select col1 from table7	6	16

To select row and Traverse cells from selected rows 
import pandas as pd 
df=pd.read_csv("/home/deepak/PycharmProjects/csv_test/test_cases.csv")
O_row=(df[df["TC_ID"]=="TC0006"])
print(O_row["TC_NAME"].values[0])
 
Output:
TC_NAME6 

To convert CSV to list of dictionaries:
 (Using Pandas)
import pandas as pd
import csv
df = pd.read_csv(sPath)
list_of_data = df.to_dict(orient='records')#dropna().to_dict(orient='recorddel df
return list_of_data
 
(Using CSV)
import pandas as pd
import csv 
with open(sPath, "r") as f:
    reader = csv.DictReader(f)
    list_dict = list(reader)
    return list_dict 

 

mac setup

Mac Setup

1. To install "brew" - Ref https://brew.sh/

Run below command:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

2. brew install ranger
3. Change terminal theme to Brew
Terminal -> Preferences -> Settings -> Click desired Profile -> Click "Default" at the bottom of the frame

4. Install sublime text from google
5. Run below command in mac

ln -s "/Applications/Sublime Text.app/Contents/SharedSupport/bin/subl" /usr/local/bin/subl

6.run command "subl" to open sublime text
7. open ranger , navigate to any text file
8. run command "!subl xx.txt"
9.alias vlc='/Applications/VLC.app/Contents/MacOS/VLC'
10. vlc input.mp4

ETL : Testing Basics

ETL : Testing Basics 

 SQL types:

• DML (Data Manuipulation) - Insert , update , Delete , Merge
• DDL (Data Definition)-Create , alter,Drop,Truncate
• DRL (Data Retrival )- Select
• TCL (Transaction Lan)- Commit Rollback
• DCL (Data Control)- grant ,revoke

Data WareHouse :

1. It is a DataBase
2. Contains data for Analysis

OLTP (Online Transaction Processing System) 

•  Used in Day to Day Transaction
• Ex : ATM , Net Banking 
• Data is Captured into either DB , Flat File (xml) etc.,

DataMart:

• It is a smaller part of DataWarehouse
• It is specific to a department eg: Finance , HR etc.,
•  It is used by middle level management to retrieve data wrt to their deptartment.

ETL ( Extract , Transform ,Load):

• Process of read data from OLTP and  inserting into Data Warehouse
• The Data from Data Warehouse is used in Analytics report

Ex: Company A has following departments Eg: Sales , Services department. 

Assume Sales departments has records of customer transaction in DB and Services has transaction in an csv file or flat file ( service number , service mechnaic name ,amount paid , spares replaced ,Date etc.,)

If management wants to find out the total transaction of a customer both sales and services how will they do ?

So they need to extract only relavent information from both data sources and load it into a common Destination .This is called ETL.

Hetrogenous Data : Data coming from different Data Sources like Flat file , Orcale DB,  Mongo, My Sql  ,txt,json etc.,

Mapping Document ( http://dwbitips.blogspot.com/2012/12/what-is-etl-mapping-document-real-time.html ) :

1. Requirement Document
2. Contains src , dest table and Column Details
3.  Conatins transformation query or forumula applied for each column

ETL testing = Data Integration Testing + BI testing (Report)

Generic Data Integration Testing : 

1. Scheme match wrt requirement
2. Count match with src and destination
3. Duplicate records (loaded twice), 
4. Null Validation
5. Untransformed Data should match source and destination data (1:1).
6. Mapping is correct ie., data is going to correct cols as per requirment
7. Old data already residing is not affected

Late Arriving Dimension or Early Arriving Fact

• Happens when you get fact data before the dimension data arrives 
• example:  New employee just onboarded and had an accident before insurance forms could be completed and processed by the insurance company. The hospital will create a medical claim record to be paid by the insurance company, but the insurance company does not yet have a person to associate the claim.
•  -1 = UNKNOWN, -2 = N/A, -3 = Not Provided etc., 

 Ref:

https://www.softwaretestinghelp.com/etl-testing-data-warehouse-testing/

https://www.guru99.com/utlimate-guide-etl-datawarehouse-testing.html

https://www.tutorialspoint.com/etl_testing/index.htm

pytest : All about pytest

pytest : All about pytest

https://drive.google.com/file/d/1-yL5U1ZQaH6NNv2XkCb2yXe_Nq0Piiw1/view?usp=sharing

Index :

1.  Installation x 2
2.  Project structure
3. Sample code
4. Important commands x 7
5. Run specific test case
6. Raise pytest exception (pytest.raises)
7. labels (pytest.mark.label_name ,pytest.mark.skip,pytest.mark.skip_if)
8. Parameterize (pytest.mark.parametrize('x,y,res',[(1,2,3),(0,3,3)]) )
9. Fixture (pytest.fixture(scope="module") )
10. conftest
11. pdb

Installation:
Using pip

pip3 install pytest

pip install selenium

Using requirements.txt : project>requirements.txt

selenium >= 3.0

pytest >= 3.10.1

pytest-html >= 1.20.0

pip install -r requirements.txt

or

pipenv install -r requirements.txt #Inside virtual env
Project structure

add file : - project>tests>test_filename.py

Sample Code

from selenium import webdriver

def test_1():

 driver=webdriver.Chrome("chromedriverpath")

 driver.get(url)

 objs=driver.find_elements_by_xpath("//button")

 for i in objs:

  i.click()

1. open terminal
2. cd to project
3. excute command "pytest"

Important commands:

1.    pytest --pdb #runs the debugger when an err is encountered

2.    pytest -s #to "print" result in console

3.    pytest -v #verbose 

4.    pytest --maxfail=2 # limit no of fails

5.    py.test --html=1.html

6.    pipenv run pytest

7.    pipenv run py.test --html=1.html

pytest -v -k "add or something" # runs only methods with add ,something

pytest -h # help

pytest -v -x #to stop after 1st fail

pytest -v -x --tb=no # do not display stack trace

pytest -q # quiet mode

pytest -rsx #report skipped tests 

pytest --lf #runs only tests that failed on last attempt

# in eclipse Window > Preferences > Pydev> PyUnit = Py.testRunner and add above Parameters 

Execute specific method of class

pytest test_filename.py::test_method

Exception
def test_mytest():

with pytest.raises(ZeroDivisionError):
  1/0

  

Markers(labels)

@pytest.mark.label_name" .To run "pytest -v -m label_name"

@pytest.mark.skip(reason="any reason") #skip methods

@pytest.mark.skip_if(sys.version_info < (3,3),reason="some reason")

Parametrize

import pytest

@pytest.mark.parametrize('x,y,res',[(1,2,3),(0,3,3)])

def test_add(x,y,res):

    assert classname.add(x,y,res)

Fixtures(similar to @Before annotations in Junit test used for - db connection , etc.,)
@pytest.fixture(scope="module") #scope =module/session/function def fix():
    pass    #setup code
    yield   #run till here
    print("done") # tear down code

def fn(fix):
    #fix runs 1st

example:
from selenium import webdriver
@pytest.fixture()
def test_setup():
    global driver
    driver =webdriver.Chrome(executable_path="C:/Users/driver.exe")# Windows
    driver.implicitly.wait(5)
    driver.maximise()
    yield()
    driver.close()
    driver.quit()

conftest.py
You can put all fixtures inside this file and this file will execute before test starts

Debugger :

import pdb;

pdb.set_trace() # to break

Others 

Eclipse Setup :

 (if u are using eclipse : Window->Preferences --> pydev --> PyUnit --> Change the Test runner to "Py.test runner".)

Right Click over the file. Run As --> Python Unit-Test

Or press Ctrl+F9:-

It will prompt you to select the test

User can use fixture or setup-teardown
setup
import pytest
def setup_module(module):

tiredown 
import pytest
def teardown_module(module):

Allure report
pip install allure-pytest
pip list | grep -i allure

pytest --alluredir=/Users/Documents/reports
allure generate /Users/Documents/reports

setup.py
#Add this to setup.py file:
from setuptools import setup
setup(
    # ...,
    setup_requires=["pytest-runner", ...],
    tests_require=["pytest", ...],
    # ...,
)

#And create an alias into setup.cfg file:
[aliases]
test=pytest
[tool:pytest]
addopts = --verbose
python_files = testing/*/*.py

#If you now type:
python setup.py test


Links 

#https://docs.pytest.org/en/latest/goodpractices.html#goodpractices

https://docs.pytest.org/en/latest/parametrize.html
https://docs.pytest.org/en/latest/example/parametrize.html#paramexamples
https://pypi.org/project/pytest-html/
https://pypi.org/project/pytest-csv/

 important read
 https://bytes.yingw787.com/posts/2018/12/10/data_driven_testing_three/
 https://gitlab.com/qalabs/blog/pytest-parametrize-example

important watch **
https://www.youtube.com/watch?v=o9pEzgHorH0
https://www.youtube.com/watch?v=RdE-d_EhzmA

hooks
https://stackoverflow.com/questions/21930858/pytest-parameterize-row-from-csv-as-a-testcase?rq=1 

Running pytest inside eclipse ide
Ref :https://stackoverflow.com/questions/37985589/how-to-debug-or-run-pytest-scripts-using-eclipse