Sunday, May 16, 2021
Saturday, May 15, 2021
Django - Simple Registration Form
Django - Simple Registration Form
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Title</title>
<p>Register</p>
<form method="POST" action="">
{% csrf_token %}
{{form.username.label}}
{{form.username}}
{{form.email.label}}
{{form.email}}
{{form.password1.label}}
{{form.password1}}
{{form.password2.label}}
{{form.password2}}
<input type="submit" name="new_user">
</form>
</head>
<body>
</body>
</html>
Monday, March 22, 2021
CORRELATED SUBQUERIES
SubQuery :
Simple subquery doesn't use values from the outer query and is being calculated only once:
SELECT id, first_name FROM student_details
WHERE id IN (SELECT student_id FROM student_subjects
WHERE subject= 'Science');
CoRelated Subquery -
Query To Find all employees whose salary is above average for their department
SELECT employee_number, name FROM employees emp
WHERE salary > ( SELECT AVG(salary) FROM employees
WHERE department = emp.department);
Sunday, March 14, 2021
Saturday, March 13, 2021
FastApi : Create Production ready Api
FastApi : Create Production ready Api (faster than Flask)
https://fastapi.tiangolo.com/
Features
1. Asynchronous
2. High Perfromance
3. Less Code
4. Data Type and Data Models auto Conversions
5. Auto Documentation
- swagger (/docs)
- ReDoc (/redoc)
Pre-Req:
Install and activate virtual environment to be safe.
Steps:
- pip install fastapi
- pip install hypercorn #server
- touch main.py
- copy paste below code
- hypercorn main:app --reload
Friday, February 5, 2021
Sunday, January 31, 2021
CIRCUITPYTHON :SEEEDUINO XIAO [SEED]
CIRCUITPYTHON :SEEEDUINO XIAO
Steps:
Uses SAMD21 Processor
1. connect SEEEDuino xiao to PC using TYPE-C cable
2. short RST pins using a cable fast , 2 times.
3. Once done successfully,Audrino drives appears
4. Go website -
https://circuitpython.org/board/seeeduino_xiao/
https://wiki.seeedstudio.com/Seeeduino-XIAO-CircuitPython/
5. Download latest .UF2 file
6. Copy and paste it inside the drive
7. Now the drive will be converted to CIRCUITPY
8. Create a file code.py
9. Copy paste below code into code.py (same for all circuit py IC)
import time
import board
from digitalio import DigitalInOut,Direction
led = DigitalInOut(board.D13) #D13 is a built in LED
#A1 - A10 can be used as well if u use a separate LED and a Resistor 100 - 400 ohms refer below for calculations
led.direction=tinker .OUTPUT
while True:
led.value = True
time.sleep(1)
led.value=False
time.sleep(1)
10. Save file
11. The LED should start blinking
A simple LED circuit consists of a LED and resistor. The resistor is used to limit the current that is being drawn and is called a current limiting resistor. Without the resistor the LED would run at too high of a voltage, resulting in too much current being drawn which in turn would instantly burn the LED, and likely also the GPIO port.
To calculate the resistor value we need to examine the specifications of the LED. Specifically we need to find the forward voltage (VF) and the forward current (IF).
A regular red LED has a
forward voltage (VF) of 1.7V
forward current of 20mA (IF).
output voltage of the IC which is 3.3V.
We can then calculate the resistor needed-
Sunday, November 29, 2020
Django
Django:
What is Django ?
Can be used to create mulitple apps like - blogs , authentication etc .,
Uses MVT architecture
MVT architecture (Model View Template)
No Controller unlike MVCBrowser > Django > urls.py > View
View pulls data from model(DB) and Displays it on Template (HTML)
How to use:
- No need to stop Django server when doing development
- There will be only 1 settings file in the main project to handle auths , templates etc.,
- When you want to create a new webpage do not modify the auto generated Framework.
- Always create your own app with the command ($python manage.py startapp app_name)
Example Source Code
- make sure python is installed.
- $pip install virtual env
- $virtualenv venv
- $source venv/bin/activate
- $pip install django
- If ur using Pycharm than goto : Preference>Select virtual Env created
- django-admin startproject Main #Start django project
- $cd Main
- python manage.py startapp PersonApp #Create App
- python manage.py runserver
- open http://127.0.0.1:8000/
- comment - settings.py > MIDDLEWARE >
- settings.py > Installed APP ['PersonApp']
- settings.py > TEMPLATES > 'DIRS': [os.path.join(BASE_DIR, 'PersonApp','html')] #import os
- Under urls.py
path('', include('PersonApp.urls'))
<p>{{ data }}</p>
<textarea name="fname">{{fname}}</textarea>
<textarea name="sname">{{sname}}</textarea>
<input type="submit">
</form>
def person(request):
if request.method == "GET":
return render(request, 'person.html', {"data":"enter ur name"})
elif request.method == "POST":
fname = request.POST.get('fname', None)
sname= request.POST.get('sname', None)
print("accepted :"+fname)
return render(request, 'person.html', {"data":"Accepted !"})
from . import views
urlpatterns= [path('',views.person,name='person')]
first_name = models.CharField(max_length=30)
last_name = models.CharField(max_length=30)
admin.site.register(Person)
python manage.py migrate
python manage.py createsuperuser
0.0.0.0/admin
def person():
person=Person()
person.first_name=fname
person.last_name=sname
person.save()
print("records ="+str(len(Person.objects.all())))
URL Dispatching
- Copy a urls.py from base Framework into the new App folder
- Notice line "path('admin/', admin.site.urls)" in Framework urls.py eg: path is /admin than redirect to admin page
- Add below line in url.py in Base Framework
- This redirects "/test" to urls.py in the App folder.
- Add below lines in urls.py in app_name :
urlpatterns = [path('test', views.test,name="test")]
- In views.py of app_name add below lines:
def test(request):
return HttpResponse("Testpage")
- Notice here controls goes from url.py(base) >> url.py >> views.py
StaticFolders:
- The files inside static folders are made public
- Can Contain CSS / JS to be used in the webpage
- Create folder "static" inside project
- create file "test.txt" with some contents inside and save
- Add this to settings.py : STATICFILES_DIRS = [BASE_DIR / "static",'/var/www/static/',]
- open link -http://127.0.0.1:8000/static/test.txt
Create Templates with paramaters
- Create folder "templates" inside project
- Create a home.html inside templates with following :
- Goto settings.py > Under TEMPLATES > Add 'DIRS': [BASE_DIR / "templates",],
- Goto to the app views.py > Add line to the related resource method -
- BootStraping
- Used for ready made websites : https://getbootstrap.com/docs/4.5/getting-started/introduction/
- Copy Starter Template html
- You can use replace the html in home.html and save
- reload page
Template Inheritance
- Inherit the base template across the whole of the application
- Create base.html in templates folder
- Add templates html here , add placeholder for replacement wherever needed
- delete contents of home.html and add :
- In first line {% extends 'base.html' %}
- Add replacements for placeholder - {% block body %} HI {% endblock body %}
- Save
Using Images in Website:
Save any images into static folderRename the extension as ".jfif"
In the html file use as "/static/file.jfif"
Admin Logging Creation(We need to create Tables)
- $python manage.py createsuperuser
- Open http://127.0.0.1:8000/admin
To Create DB Table:
API
Pre-Req
- pip install djangorestframework
- settings.py > INSTALLED_APPS = 'rest_framework'
- url.py > urlpatterns=[path('api/', include('api.urls')),]
pip freeze > requirements.txt
python manage.py startapp api
create api/serializers.py : Converts models to Json
from .models import Person
class personSerializer(serializers.HyperlinkedModelSerializer):
class Meta:
model = Person
fields = ("url","first_name","last_name")
api/views.py
from .models import Person
from .serializers import personSerializer
#make sure Person model is created in models.py and migrated
class personView(viewsets.ModelViewSet):
queryset = Person.objects.all()
serializer_class = personSerializer
create api/url.py : Handles all API endpoint
from . import views
from rest_framework import routers
router = routers.DefaultRouter()
router.register('person',views.personview)
urlpatterns = [path('', include(router.urls))]
Ref:
https://www.youtube.com/watch?v=JxzZxdht-XY&t=5368sPython Virtual Environment (venv)
Python Virtual Environment (venv)
Pre-Req :
- This is on Mac
- Make sure Python3 is installed in the system
- $pip3 install virtualenv #--upgrade
- $virtualenv #help
Steps:
- create folder
- cd inside folderName
- $virtualenv env_name #-p /folder_path_of_python_to_clone/bin/python
- $source ./env_name/bin/activate
- #if u get permission denied "$chmod -R 777 *"
- #Note u can also have multiple environments and activate which ever u want
- Notice "env_name" in the Terminal line meaning it is activated.
- Try:
- $python
- import requests # you will get error
- exit()
- Try2 (make sure virtual env is still activated)
- $pip install requests
- $python
- import requests
- exit()
- $pip freeze > requirements.txt
- To install requirements from above file #pip install -r requirements.txt
- $deactivate #Deactivate the env
Note:
- To use Virtual Environment in Pycharm
- Preferences> Settings > Add > Virtualenv
- Make sure it is pointing to venv
Tuesday, November 24, 2020
Sunday, November 22, 2020
Nginx and Gunicorn
Nginx and Gunicorn
NGINX- Serving Static Content
- Reverse Proxy (forward requests from clients to backend servers)
- load balancing / distributing traffic across multiple servers.
- API Gateway (single entry point for all API requests)
- Caching ( caching frequently requested content)
- Nginx sits in front of Gunicorn/ unicorn.
- Gunicorn (uWSGI can be used as well) serves your flask app
- For example, if the incoming request is an http request Nginx redirects it to gunicorn, if it is for a static file, it serves it itself.
Example
Pre-Req:
- Docker is installed
- Create Test Folder with below files:
- Dockerfile
- requirments.txt
- flask.py
FROM python:3
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD ["gunicorn", "--workers=1", "--bind=0.0.0.0:5000", "flask:app"]
------requirments.txt------
flask==1.1.1
gunicorn
------flask.py------
from flask import Flask, request
app = Flask(__name__)
@app.route('/')
def hello_world() -> str: return "Hello world"
if __name__ == '__main__':app.run(debug=True, host='0.0.0.0')
------------------------------------
Steps:
- $ cd to Test Folder
- $ docker run -it --rm -p 5000:5000 -v ${PWD}:/app python:3 bash # cd /app
- cd /apps
- pip install -r requirements.txt
- export FLASK_APP=flask.py
- gunicorn --bind=0.0.0.0:5000 --workers=1 flask:app
Ref
https://stackoverflow.com/questions/43044659/what-is-the-purpose-of-using-nginx-with-gunicorn
https://rahmonov.me/posts/run-a-django-app-with-gunicorn-in-ubuntu-16-04/
Saturday, November 21, 2020
Login API with Jmeter and Python Request:
Login API with Jmeter and Python Request:
Scenario:
- To record a login API transaction using Jmeter
- Run the same using Jmeter and check the response
- Reproduce the same using Jmeter
Pre-Req:
- FireFox Browser
- Jmeter
- Python 3.7+
- pip install requests
Settings:
Configure Port numbers
- Inorder for Jmeter to capture recording , we need to place Jmeter in-between Browser and external connections
- such that all data passes through Jmeter before reaching the Browser and vis versa.
- This is done by changing the port number of Browser to use port number of Jmeter. So that Jmeter can listen to the interactions
Browser -----------> JMETER -------------> External Connections
- open Jmeter
- File > templates > Select Recording >Create
- In the folder structure > copy the port name
- Open firefox > Preferences>
- Scroll to bottom > Network Settings >Click on Settings
- Select Manul proxy config
- Http proxy = localhost and port = Paste the port number
- click on OK
Installing Certificates :
- In Jmeter , click on Https test Script Recorder
- Click on Start button
- You will get a message box saying a root CA certificate is created under "Jmeter bin Directory"
- Open FireFox preferences > Privacy and Security
- View Certificates
- Click on import
- Select the certificate file created under "Jmeter/bin"
- Click on Ok
- Close Preferences
- stop the recording in Jmeter
Steps to record Capture in Jmeter:
- Start recording in Jmeter
- Open FireFox
- open link https://login.shrm.org/
- Enter email -abc@gmail.com , password = 123
- click on Sign in
- Stop recording in Jmeter
- Under Jmeter open Thread Group> Recording Controller and Notice the recording
- under Jmeter open Https Test Script Recorder > View Results Tree and Check the response as well
- Toggle between Http , Raw Tabs and Sample Result , Request and Response Tabs to understand the Request method and how data is send and received
- Close the browser
- Now you can run the same for different username and password by changing values in Recording Controller > Http Header manager
Steps to Reproduce in Python
- Open Terminal
- $python3
url="https://www.login.shrm.org/"
headers={'Content-Type': 'application/json'}
data={'LoginCredentials.Username': 'deep@xx.com', 'LoginCredential.Password': '123'}
r= requests.post(url,headers=headers,data=data)
print(r.status_code)
print(r.content)
Friday, November 20, 2020
Spark Streaming
Spark Streaming (Scala)
Steps :1. Open Terminal
2. nc -l 54321 #any port number given in the program
3. Run below code
4. Type anything in terminal
5. Notice the Code Output
build.sbt
version := "0.1"
scalaVersion := "2.11.0"
val sparkVersion = "2.4.7"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % sparkVersion,
"org.apache.spark" %% "spark-mllib" % sparkVersion,
"org.apache.spark" %% "spark-sql" % sparkVersion,
"org.apache.spark" %% "spark-streaming-flume" % "2.4.7"
)
libraryDependencies += "mysql" % "mysql-connector-java" % "5.1.24"
StreamingNetworkWordCount.scala
import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}
object StreamingNetworkWordCount {
def main(args:Array[String]) {
val sparkConf = new SparkConf().setAppName("NetworkWordCount").setMaster("local[2]")
val ssc = new StreamingContext(sparkConf, Seconds(10))
val lines = ssc.socketTextStream("localhost",54321)
val words = lines.flatMap(_.split(" "))
val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
wordCounts.print()
ssc.start() // Start the computation
ssc.awaitTermination() // Wait for the computation to terminate
}
}
Sunday, November 15, 2020
Monday, November 9, 2020
AirFlow
AIRFLOW
Pre-Req:
Method 1
Method 2
- pip3 install apache-airflow
- $airflow version
- $airflow initdb
- Open New Terminal Window
- $airflow webserver
- Open new Terminal Window
- $airflow scheduler
- In browser - localhost:8080
- By default all Dags are stored in example_dags (/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airflow/example_dags)
- Automtically this file should appear in Web UI
- run the Task
Notes:
- pip3 install attr
- pip3 install cattrs
- Append this to your ~/.bash_profile:
- export LC_ALL=en_US.UTF-8
- export LANG=en_US.UTF-8
- source ~/.bash_profile
- sudo lsof -i tcp:8080
- kill -9 PID#all PIDs
- rm -rf airflow-webserver.pid
- https://stackoverflow.com/a/59086513
Tuesday, July 21, 2020
Scala : Generic Types in Scala
Generic Types in Scala
Explanation:
Sunday, July 12, 2020
Test Drive Development (TDD)
- Red-Green Refractor ie., Write Failing test First and later test should pass as the development progress
- Canary Test = Blank test to validate if the environment is working
- Description / Console output / Naming should read like a book
- Pending tests ie., Only declared but not implemented
- Error message from the Unit test should drive the development
- Source Code should be separate from the Unit Test (As it does not need to go to production. It can be in the same repo not the same file)
- The method under test initially should return below as the development progress :
- "NULL"
- Change "NULL" > Constant
- Constant > "Complex Expression depending on the requirement"
- Pass the tests
- Reveal Intention
- No Duplication
- Fewest number of elements
- Always Unit test should come first than we start the actual code.
- List all the test initially with only declaration ie.,(only describe and it()) these will be considered as pending tests or todo list.
- 1st test in canary test (No test , pass by default)
- Meaningful variable and methods names which reveal intention in tests /Actual Code.
- Comments are unnecessary if you have meaningful names.
- Code should not be duplicated
- should be < 1
- use functions /methods for repetitions.
- Make Sure all the tests are declared initially
- And only Canary Test has test definition.
- Write a Unit test which sends null / blank as input to the function under test.
- Modify the Actual code to handle null / blank
- Run the test it should pass.
- Write a test / Add definition to empty test declaration which sends a simple constant as input to Actual Code
- Modify the Actual code to handle that constant either by returning it directly / other means and make sure both the test passes
- Run the test it should pass both the tests.
- Add definition to next empty test declaration to send little bit complex input.
- Modify the Actual code to handle that constant either by using if statements
- Run the test it should pass all 3 tests.
- Add definition to next empty test declaration to send even more complex input to actual code
- Modify the Actual code to handle that constant either by using better logic and so on.
- .....