Monday, May 27, 2019

pandas : Lambda ,Map and Eval Conditions

Lambda ,Map and eval in Pandas


1. Simple 


import pandas as pd

a=pd.DataFrame({"a":["deea","123","deep","deepak"]})
b=pd.DataFrame({"b":["deeap","1234","deepak","deepa"]})
c=pd.DataFrame(map(lambda x,y:x in y,a['a'],b['b']))
print(c)
 
 
result : 
       0
0   True
1   True
2   True
3  False 
 

2. Using Dynamic Formula :

import pandas as pd

a=pd.DataFrame({"a":["deea","123","deep","deepak"]})
b=pd.DataFrame({"b":["deeap","1234","deepak","deepa"]})
c=pd.DataFrame(map(eval(input("enter -lambda x,y:x in y")),a['a'].values,b['b'].values))
print(c)

       0
0   True
1   True
2   True
3  False
 

3)Using Dynamic Formula and 1st columns

import pandas as pd

a=pd.DataFrame({"a":["deea","123","deep","deepak"]})
b=pd.DataFrame({"b":["deeap","1234","deepak","deepa"]})
c=pd.DataFrame(map(eval(input("enter -lambda x,y:x in y")),a.iloc[:,0],b.iloc[:,0]))
print(c)
  
Result: 
  0
0 True
1 True
2 True
3 False 
 
"""

"""

Column Names

data.iloc[:,0] # first column of data frame (first_name)
data.iloc[:,1] # second column of data frame (last_name)
data.iloc[:,-1] # last column of data frame (id)

Rows and Columns
data.iloc[0:5] # first five rows of dataframe
data.iloc[:, 0:2] # first two columns of data frame with all rows
data.iloc[[0,3,6,24], [0,5,6]] # 1st, 4th, 7th, 25th row + 1st 6th 7th columns.
data.iloc[0:5, 5:8] # first 5 rows and 5th, 6th, 7th columns of data frame
"""

Thursday, May 23, 2019

Numpy : Lambda / map / vectorize


Numpy : Lambda / map / vectorize

import numpy as np

def myfunc(a, b):
    if a > b:
        return a - b
    else:
        return a + b

n1=np.array([1,2,3])
n2=np.array([5,4,3])

vfunc = np.vectorize(myfunc)
print(vfunc(n1,n2))
Output :
[6 6 6]

Saturday, May 4, 2019

pandas : Import snippet

pandas : Import snippet






TC_ID TC_NAME QUERY1 QUERY2 CONDITION BASICTEST(QUERY1/QUERY2)
TC0001 TC_NAME1 Select col1 from table1 Select col1 from table2 1 11
TC0002 TC_NAME2 Select col1 from table2 Select col1 from table3 2 12
TC0003 TC_NAME3 Select col1 from table3 Select col1 from table4 3 13
TC0004 TC_NAME4 Select col1 from table4 Select col1 from table5 4 14
TC0005 TC_NAME5 Select col1 from table5 Select col1 from table6 5 15
TC0006 TC_NAME6 Select col1 from table6 Select col1 from table7 6 16



To select row and Traverse cells from selected rows 
import pandas as pd 
df=pd.read_csv("/home/deepak/PycharmProjects/csv_test/test_cases.csv")
O_row=(df[df["TC_ID"]=="TC0006"])
print(O_row["TC_NAME"].values[0])
 
Output:
TC_NAME6 

To convert CSV to list of dictionaries:
 (Using Pandas)
import pandas as pd
import csv
df = pd.read_csv(sPath)
list_of_data = df.to_dict(orient='records')#dropna().to_dict(orient='recorddel df
return list_of_data
 
(Using CSV)
import pandas as pd
import csv 
with open(sPath, "r") as f:
    reader = csv.DictReader(f)
    list_dict = list(reader)
    return list_dict 

 

Saturday, April 27, 2019

mac setup

Mac Setup



1. To install "brew" - Ref https://brew.sh/
Run below command:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

2. brew install ranger
3. Change terminal theme to Brew
Terminal -> Preferences -> Settings -> Click desired Profile -> Click "Default" at the bottom of the frame

4. Install sublime text from google
5. Run below command in mac

ln -s "/Applications/Sublime Text.app/Contents/SharedSupport/bin/subl" /usr/local/bin/subl

6.run command "subl" to open sublime text
7. open ranger , navigate to any text file
8. run command "!subl xx.txt"
9.alias vlc='/Applications/VLC.app/Contents/MacOS/VLC'
10. vlc input.mp4

Wednesday, April 24, 2019

ETL : Testing Basics

ETL : Testing Basics 

 SQL types:
  • DML (Data Manuipulation) - Insert , update , Delete , Merge
  • DDL (Data Definition)-Create , alter,Drop,Truncate
  • DRL (Data Retrival )- Select
  • TCL (Transaction Lan)- Commit Rollback
  • DCL (Data Control)- grant ,revoke
Data WareHouse :
  1. It is a DataBase
  2. Contains data for Analysis
OLTP (Online Transaction Processing System
  •  Used in Day to Day Transaction
  • Ex : ATM , Net Banking 
  • Data is Captured into either DB , Flat File (xml) etc.,
DataMart:
  • It is a smaller part of DataWarehouse
  • It is specific to a department eg: Finance , HR etc.,
  •  It is used by middle level management to retrieve data wrt to their deptartment.
ETL ( Extract , Transform ,Load):
  • Process of read data from OLTP and  inserting into Data Warehouse
  • The Data from Data Warehouse is used in Analytics report
Ex: Company A has following departments Eg: Sales , Services department. 
Assume Sales departments has records of customer transaction in DB and Services has transaction in an csv file or flat file ( service number , service mechnaic name ,amount paid , spares replaced ,Date etc.,)
If management wants to find out the total transaction of a customer both sales and services how will they do ?
So they need to extract only relavent information from both data sources and load it into a common Destination .This is called ETL.
Hetrogenous Data : Data coming from different Data Sources like Flat file , Orcale DB,  Mongo, My Sql  ,txt,json etc.,
Mapping Document ( http://dwbitips.blogspot.com/2012/12/what-is-etl-mapping-document-real-time.html ) :
  1. Requirement Document
  2. Contains src , dest table and Column Details
  3.  Conatins transformation query or forumula applied for each column

ETL testing = Data Integration Testing + BI testing (Report)
Generic Data Integration Testing : 
  1. Scheme match wrt requirement
  2. Count match with src and destination
  3. Duplicate records (loaded twice), 
  4. Null Validation
  5. Untransformed Data should match source and destination data (1:1).
  6. Mapping is correct ie., data is going to correct cols as per requirment
  7. Old data already residing is not affected

Late Arriving Dimension or Early Arriving Fact

  • Happens when you get fact data before the dimension data arrives 
  • example:  New employee just onboarded and had an accident before insurance forms could be completed and processed by the insurance company. The hospital will create a medical claim record to be paid by the insurance company, but the insurance company does not yet have a person to associate the claim.
  •  -1 = UNKNOWN, -2 = N/A, -3 = Not Provided etc., 



 Ref:
https://www.softwaretestinghelp.com/etl-testing-data-warehouse-testing/
https://www.guru99.com/utlimate-guide-etl-datawarehouse-testing.html
https://www.tutorialspoint.com/etl_testing/index.htm

Sunday, April 21, 2019

pytest : All about pytest

pytest : All about pytest




Index :
  1.  Installation x 2
  2.  Project structure
  3. Sample code
  4. Important commands x 7
  5. Run specific test case
  6. Raise pytest exception (pytest.raises)
  7. labels (pytest.mark.label_name ,pytest.mark.skip,pytest.mark.skip_if)
  8. Parameterize (pytest.mark.parametrize('x,y,res',[(1,2,3),(0,3,3)]) )
  9. Fixture (pytest.fixture(scope="module") )
  10. conftest
  11. pdb
Installation:
Using pip
pip3 install pytest
pip install selenium

Using requirements.txt : project>requirements.txt
selenium >= 3.0
pytest >= 3.10.1
pytest-html >= 1.20.0

pip install -r requirements.txt
or
pipenv install -r requirements.txt #Inside virtual env
Project structure

add file : - project>tests>test_filename.py

Sample Code
from selenium import webdriver
def test_1():
 driver=webdriver.Chrome("chromedriverpath")
 driver.get(url)
 objs=driver.find_elements_by_xpath("//button")
 for i in objs:
  i.click()
  1. open terminal
  2. cd to project
  3. excute command "pytest"
Important commands:
1.    pytest --pdb #runs the debugger when an err is encountered
2.    pytest -s #to "print" result in console
3.    pytest -v #verbose 
4.    pytest --maxfail=2 # limit no of fails
5.    py.test --html=1.html
6.    pipenv run pytest
7.    pipenv run py.test --html=1.html

pytest -v -k "add or something" # runs only methods with add ,something
pytest -h # help
pytest -v -x #to stop after 1st fail
pytest -v -x --tb=no # do not display stack trace
pytest -q # quiet mode
pytest -rsx #report skipped tests 
pytest --lf #runs only tests that failed on last attempt
# in eclipse Window > Preferences > Pydev> PyUnit = Py.testRunner and add above Parameters 

Execute specific method of class
pytest test_filename.py::test_method

Exception
def test_mytest():
with pytest.raises(ZeroDivisionError):
  1/0
  
Markers(labels)
@pytest.mark.label_name" .To run "pytest -v -m label_name"
@pytest.mark.skip(reason="any reason") #skip methods
@pytest.mark.skip_if(sys.version_info < (3,3),reason="some reason")

Parametrize
import pytest
@pytest.mark.parametrize('x,y,res',[(1,2,3),(0,3,3)])
def test_add(x,y,res):
    assert classname.add(x,y,res)

Fixtures(similar to @Before annotations in Junit test used for - db connection , etc.,)

@pytest.fixture(scope="module") #scope =module/session/function def fix():
    pass    #setup code
    yield   #run till here
    print("done") # tear down code

def fn(fix):
    #fix runs 1st

example:
from selenium import webdriver
@pytest.fixture()
def test_setup():
    global driver
    driver =webdriver.Chrome(executable_path="C:/Users/driver.exe")# Windows
    driver.implicitly.wait(5)
    driver.maximise()
    yield()

    driver.close()
    driver.quit()

conftest.py
You can put all fixtures inside this file and this file will execute before test starts


Debugger :
import pdb;
pdb.set_trace() # to break

Others 

Eclipse Setup :
 (if u are using eclipse : Window->Preferences --> pydev --> PyUnit --> Change the Test runner to "Py.test runner".)
Right Click over the file. Run As --> Python Unit-Test
Or press Ctrl+F9:-
It will prompt you to select the test

User can use fixture or setup-teardown
setup
import pytest
def setup_module(module):


tiredown
 
import pytest
def teardown_module(module):

Allure report
pip install allure-pytest
pip list | grep -i allure


pytest --alluredir=/Users/Documents/reports
allure generate /Users/Documents/reports

setup.py
#Add this to setup.py file:
from setuptools import setup
setup(
    # ...,
    setup_requires=["pytest-runner", ...],
    tests_require=["pytest", ...],
    # ...,
)


#And create an alias into setup.cfg file:
[aliases]
test=pytest
[tool:pytest]
addopts = --verbose
python_files = testing/*/*.py


#If you now type:
python setup.py test



Links 
#https://docs.pytest.org/en/latest/goodpractices.html#goodpractices
https://docs.pytest.org/en/latest/parametrize.html
https://docs.pytest.org/en/latest/example/parametrize.html#paramexamples
https://pypi.org/project/pytest-html/
https://pypi.org/project/pytest-csv/

 important read
 https://bytes.yingw787.com/posts/2018/12/10/data_driven_testing_three/
 https://gitlab.com/qalabs/blog/pytest-parametrize-example

important watch **
https://www.youtube.com/watch?v=o9pEzgHorH0
https://www.youtube.com/watch?v=RdE-d_EhzmA

hooks
https://stackoverflow.com/questions/21930858/pytest-parameterize-row-from-csv-as-a-testcase?rq=1 

Running pytest inside eclipse ide
Ref :https://stackoverflow.com/questions/37985589/how-to-debug-or-run-pytest-scripts-using-eclipse


Friday, April 5, 2019

mac : Adding custom paths

Adding custom paths


Permanently

>nano ~/.profile
any_name="/path"
save and close
>$any_name



Temporary (only for the current session)

>alias any_Name="cd /path"
>any_Name