MongoDb : MongoDb no SQL

pandas : Lambda ,Map and Eval Conditions

Lambda ,Map and eval in Pandas

1. Simple 

import pandas as pd

a=pd.DataFrame({"a":["deea","123","deep","deepak"]})

b=pd.DataFrame({"b":["deeap","1234","deepak","deepa"]})

c=pd.DataFrame(map(lambda x,y:x in y,a['a'],b['b']))

print(c)

 

 

result : 

       0

0   True

1   True

2   True

3  False 

 

2. Using Dynamic Formula :

import pandas as pd

a=pd.DataFrame({"a":["deea","123","deep","deepak"]})

b=pd.DataFrame({"b":["deeap","1234","deepak","deepa"]})

c=pd.DataFrame(map(eval(input("enter -lambda x,y:x in y")),a['a'].values,b['b'].values))

print(c)

       0

0   True

1   True

2   True

3  False

 

3)Using Dynamic Formula and 1st columns

import pandas as pd

a=pd.DataFrame({"a":["deea","123","deep","deepak"]})

b=pd.DataFrame({"b":["deeap","1234","deepak","deepa"]})

c=pd.DataFrame(map(eval(input("enter -lambda x,y:x in y")),a.iloc[:,0],b.iloc[:,0]))

print(c)

  

Result: 

  0

0 True

1 True

2 True

3 False 

 

"""

"""

Column Names

data.iloc[:,0] # first column of data frame (first_name)
data.iloc[:,1] # second column of data frame (last_name)
data.iloc[:,-1] # last column of data frame (id)

Rows and Columns
data.iloc[0:5] # first five rows of dataframe
data.iloc[:, 0:2] # first two columns of data frame with all rows
data.iloc[[0,3,6,24], [0,5,6]] # 1st, 4th, 7th, 25th row + 1st 6th 7th columns.
data.iloc[0:5, 5:8] # first 5 rows and 5th, 6th, 7th columns of data frame
"""

Numpy : Lambda / map / vectorize

Numpy : Lambda / map / vectorize

import numpy as np

def myfunc(a, b):

    if a > b:

        return a - b

    else:

        return a + b

n1=np.array([1,2,3])

n2=np.array([5,4,3])

vfunc = np.vectorize(myfunc)

print(vfunc(n1,n2))

Output :

[6 6 6]

pandas : Import snippet

pandas : Import snippet

TC_ID	TC_NAME	QUERY1	QUERY2	CONDITION	BASICTEST(QUERY1/QUERY2)
TC0001	TC_NAME1	Select col1 from table1	Select col1 from table2	1	11
TC0002	TC_NAME2	Select col1 from table2	Select col1 from table3	2	12
TC0003	TC_NAME3	Select col1 from table3	Select col1 from table4	3	13
TC0004	TC_NAME4	Select col1 from table4	Select col1 from table5	4	14
TC0005	TC_NAME5	Select col1 from table5	Select col1 from table6	5	15
TC0006	TC_NAME6	Select col1 from table6	Select col1 from table7	6	16

To select row and Traverse cells from selected rows 
import pandas as pd 
df=pd.read_csv("/home/deepak/PycharmProjects/csv_test/test_cases.csv")
O_row=(df[df["TC_ID"]=="TC0006"])
print(O_row["TC_NAME"].values[0])
 
Output:
TC_NAME6 

To convert CSV to list of dictionaries:
 (Using Pandas)
import pandas as pd
import csv
df = pd.read_csv(sPath)
list_of_data = df.to_dict(orient='records')#dropna().to_dict(orient='recorddel df
return list_of_data
 
(Using CSV)
import pandas as pd
import csv 
with open(sPath, "r") as f:
    reader = csv.DictReader(f)
    list_dict = list(reader)
    return list_dict 

 

mac setup

Mac Setup

1. To install "brew" - Ref https://brew.sh/

Run below command:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

2. brew install ranger
3. Change terminal theme to Brew
Terminal -> Preferences -> Settings -> Click desired Profile -> Click "Default" at the bottom of the frame

4. Install sublime text from google
5. Run below command in mac

ln -s "/Applications/Sublime Text.app/Contents/SharedSupport/bin/subl" /usr/local/bin/subl

6.run command "subl" to open sublime text
7. open ranger , navigate to any text file
8. run command "!subl xx.txt"
9.alias vlc='/Applications/VLC.app/Contents/MacOS/VLC'
10. vlc input.mp4

ETL : Testing Basics

ETL : Testing Basics 

 SQL types:

• DML (Data Manuipulation) - Insert , update , Delete , Merge
• DDL (Data Definition)-Create , alter,Drop,Truncate
• DRL (Data Retrival )- Select
• TCL (Transaction Lan)- Commit Rollback
• DCL (Data Control)- grant ,revoke

Data WareHouse :

1. It is a DataBase
2. Contains data for Analysis

OLTP (Online Transaction Processing System) 

•  Used in Day to Day Transaction
• Ex : ATM , Net Banking 
• Data is Captured into either DB , Flat File (xml) etc.,

DataMart:

• It is a smaller part of DataWarehouse
• It is specific to a department eg: Finance , HR etc.,
•  It is used by middle level management to retrieve data wrt to their deptartment.

ETL ( Extract , Transform ,Load):

• Process of read data from OLTP and  inserting into Data Warehouse
• The Data from Data Warehouse is used in Analytics report

Ex: Company A has following departments Eg: Sales , Services department. 

Assume Sales departments has records of customer transaction in DB and Services has transaction in an csv file or flat file ( service number , service mechnaic name ,amount paid , spares replaced ,Date etc.,)

If management wants to find out the total transaction of a customer both sales and services how will they do ?

So they need to extract only relavent information from both data sources and load it into a common Destination .This is called ETL.

Hetrogenous Data : Data coming from different Data Sources like Flat file , Orcale DB,  Mongo, My Sql  ,txt,json etc.,

Mapping Document ( http://dwbitips.blogspot.com/2012/12/what-is-etl-mapping-document-real-time.html ) :

1. Requirement Document
2. Contains src , dest table and Column Details
3.  Conatins transformation query or forumula applied for each column

ETL testing = Data Integration Testing + BI testing (Report)

Generic Data Integration Testing : 

1. Scheme match wrt requirement
2. Count match with src and destination
3. Duplicate records (loaded twice), 
4. Null Validation
5. Untransformed Data should match source and destination data (1:1).
6. Mapping is correct ie., data is going to correct cols as per requirment
7. Old data already residing is not affected

Late Arriving Dimension or Early Arriving Fact

• Happens when you get fact data before the dimension data arrives 
• example:  New employee just onboarded and had an accident before insurance forms could be completed and processed by the insurance company. The hospital will create a medical claim record to be paid by the insurance company, but the insurance company does not yet have a person to associate the claim.
•  -1 = UNKNOWN, -2 = N/A, -3 = Not Provided etc., 

 Ref:

https://www.softwaretestinghelp.com/etl-testing-data-warehouse-testing/

https://www.guru99.com/utlimate-guide-etl-datawarehouse-testing.html

https://www.tutorialspoint.com/etl_testing/index.htm

pytest : All about pytest

pytest : All about pytest

https://drive.google.com/file/d/1-yL5U1ZQaH6NNv2XkCb2yXe_Nq0Piiw1/view?usp=sharing

Index :

1.  Installation x 2
2.  Project structure
3. Sample code
4. Important commands x 7
5. Run specific test case
6. Raise pytest exception (pytest.raises)
7. labels (pytest.mark.label_name ,pytest.mark.skip,pytest.mark.skip_if)
8. Parameterize (pytest.mark.parametrize('x,y,res',[(1,2,3),(0,3,3)]) )
9. Fixture (pytest.fixture(scope="module") )
10. conftest
11. pdb

Installation:
Using pip

pip3 install pytest

pip install selenium

Using requirements.txt : project>requirements.txt

selenium >= 3.0

pytest >= 3.10.1

pytest-html >= 1.20.0

pip install -r requirements.txt

or

pipenv install -r requirements.txt #Inside virtual env
Project structure

add file : - project>tests>test_filename.py

Sample Code

from selenium import webdriver

def test_1():

 driver=webdriver.Chrome("chromedriverpath")

 driver.get(url)

 objs=driver.find_elements_by_xpath("//button")

 for i in objs:

  i.click()

1. open terminal
2. cd to project
3. excute command "pytest"

Important commands:

1.    pytest --pdb #runs the debugger when an err is encountered

2.    pytest -s #to "print" result in console

3.    pytest -v #verbose 

4.    pytest --maxfail=2 # limit no of fails

5.    py.test --html=1.html

6.    pipenv run pytest

7.    pipenv run py.test --html=1.html

pytest -v -k "add or something" # runs only methods with add ,something

pytest -h # help

pytest -v -x #to stop after 1st fail

pytest -v -x --tb=no # do not display stack trace

pytest -q # quiet mode

pytest -rsx #report skipped tests 

pytest --lf #runs only tests that failed on last attempt

# in eclipse Window > Preferences > Pydev> PyUnit = Py.testRunner and add above Parameters 

Execute specific method of class

pytest test_filename.py::test_method

Exception
def test_mytest():

with pytest.raises(ZeroDivisionError):
  1/0

  

Markers(labels)

@pytest.mark.label_name" .To run "pytest -v -m label_name"

@pytest.mark.skip(reason="any reason") #skip methods

@pytest.mark.skip_if(sys.version_info < (3,3),reason="some reason")

Parametrize

import pytest

@pytest.mark.parametrize('x,y,res',[(1,2,3),(0,3,3)])

def test_add(x,y,res):

    assert classname.add(x,y,res)

Fixtures(similar to @Before annotations in Junit test used for - db connection , etc.,)
@pytest.fixture(scope="module") #scope =module/session/function def fix():
    pass    #setup code
    yield   #run till here
    print("done") # tear down code

def fn(fix):
    #fix runs 1st

example:
from selenium import webdriver
@pytest.fixture()
def test_setup():
    global driver
    driver =webdriver.Chrome(executable_path="C:/Users/driver.exe")# Windows
    driver.implicitly.wait(5)
    driver.maximise()
    yield()
    driver.close()
    driver.quit()

conftest.py
You can put all fixtures inside this file and this file will execute before test starts

Debugger :

import pdb;

pdb.set_trace() # to break

Others 

Eclipse Setup :

 (if u are using eclipse : Window->Preferences --> pydev --> PyUnit --> Change the Test runner to "Py.test runner".)

Right Click over the file. Run As --> Python Unit-Test

Or press Ctrl+F9:-

It will prompt you to select the test

User can use fixture or setup-teardown
setup
import pytest
def setup_module(module):

tiredown 
import pytest
def teardown_module(module):

Allure report
pip install allure-pytest
pip list | grep -i allure

pytest --alluredir=/Users/Documents/reports
allure generate /Users/Documents/reports

setup.py
#Add this to setup.py file:
from setuptools import setup
setup(
    # ...,
    setup_requires=["pytest-runner", ...],
    tests_require=["pytest", ...],
    # ...,
)

#And create an alias into setup.cfg file:
[aliases]
test=pytest
[tool:pytest]
addopts = --verbose
python_files = testing/*/*.py

#If you now type:
python setup.py test


Links 

#https://docs.pytest.org/en/latest/goodpractices.html#goodpractices

https://docs.pytest.org/en/latest/parametrize.html
https://docs.pytest.org/en/latest/example/parametrize.html#paramexamples
https://pypi.org/project/pytest-html/
https://pypi.org/project/pytest-csv/

 important read
 https://bytes.yingw787.com/posts/2018/12/10/data_driven_testing_three/
 https://gitlab.com/qalabs/blog/pytest-parametrize-example

important watch **
https://www.youtube.com/watch?v=o9pEzgHorH0
https://www.youtube.com/watch?v=RdE-d_EhzmA

hooks
https://stackoverflow.com/questions/21930858/pytest-parameterize-row-from-csv-as-a-testcase?rq=1 

Running pytest inside eclipse ide
Ref :https://stackoverflow.com/questions/37985589/how-to-debug-or-run-pytest-scripts-using-eclipse