4) An Overview on Logistic Regression in ML

 An Overview on Logistic Regression in ML

Introduction:

                               This is a supervised ML algorithm and also known to solve the binary classification problems. Binary means two and classification means natures and this means that we use it, to predict the nature between two natures.  In simple words here we have two classes it can be true(presence) or false(absence) or may be 1 or 0,  Like Linear Regression problem which we discussed in blog 4, we give and input x and model (discuss it later) will predict output y. In of the point is false categories called negative class and true categories called positive class.

How to achieve Logistic Regression:

                             To Achieve logistic regression we use sigmoid function which gives us the value of 0 and 1. We apply condition here to get the prediction in 0 and 1 form like if f(x) which is output greater than 0.5 then real output will be 1 and if less than 0.5 then real output will be 0. This we cannot achieve if we start using the linear function for this kind of problem also like linear regression problems because linear function can give us the output in wide variety of numbers which can make difficult to get classification number like 0 and 1. 

Sigmoid Formula:

               

An overview of Logistic Regression with single neuron:

As you can see x is input which will multiply by weight(w) and add by bias(b) and sigma is activation

function sigmoid after applying sigmoid we will get the output y. Here after getting the output y we will compare thus output with the actual output if both are same or close then good we test this thing using the loss function if loss is less then good otherwise we will change the value of weights and bias using gradient decent. This we will do until we reach the closest value.

Loss Function Formula:

                         L(f w[ i ] ,b ) ,y1 ) =- ( yi ) log ( fwb ( x[i] ) ) - ( 1 - yi ) log ( 1 - fw[i] , b ( x[i] ) )

Cost Function Formula (Gradient Decent Formula):

                         J ( w[i] , b ) = 1 / m ( sigma ( L ( fw [i] , b ( x[i] , yi ) ) )

Notes:

Code:

from distutils.log import warn
import numpy as np
import matplotlib.pyplot as py
x_train=np.array([2,3,5,6,9,1])
y_train=np.array([0,0,1,1,1,0])
np.seterr(divide = 'ignore') 
# np.seterr(divide='warn')
def calculate(x,w,b):
    fx=np.dot(x,w)+b
    return fx
def sigmoid(z):
    fx=1/(1+2.71**(-z))
    return fx
def cost(x,y,w,b):
    m=len(x)
    t_cost=0
    sum_cost=0
    for i in range(m):
        fx=sigmoid(np.dot(x[i],w)+b)
        print("here is the issue ",fx)
        t_cost=-(x[i]*np.log(fx))-((1-x[i])*np.log(1-fx))
        sum_cost+=t_cost
    total_cost=(1/m)*sum_cost
    return total_cost
def gradientLoss(x,y,w,b):
    m=len(x)
    lossW=0
    lossB=0
    sum_loss_w=0
    sum_loss_b=0
    historyW=np.zeros(m)
    historyB=np.zeros(m)
    for i in range(m):
        fx=sigmoid(np.dot(x[i],w)+b)
        loss=fx-y[i]
        lossW=loss*x[i]
        lossB=loss
        historyW[i]=lossW
        historyB[i]=lossB
        sum_loss_w+=lossW
        sum_loss_b+=lossB
    sum_loss_w=(1/m)*sum_loss_w
    sum_loss_b=(1/m)*sum_loss_b
    return sum_loss_w,sum_loss_b

def gradient(x,y,w,b,lr,iterations):
    historyW=np.zeros(iterations)
    historyB=np.zeros(iterations)
    for i in range(iterations):
        gradientW,gradientB=gradientLoss(x,y,w,b)
        w=w-(lr*(gradientW))
        b=b-(lr*(gradientB))
        historyW[i]=w
        historyB[i]=b
    return historyW,historyB
def predict(x,w,b):
    result=np.zeros(len(x))
    for i in range(len(x)):
        output=calculate(x_train[i],w,b)
        if(output>0):
            result[i]=1
        else:
            result[1]=0
    return result    
        
    
w=0.1
b=1
lr=0.1
historyW,historyB=gradient(x_train,y_train,w,b,lr,1000)
print("Weights are ",historyW)
print("Bias are ",historyB)
w=historyW[-1]
b=historyB[-1]
# total_cost=cost(x_train,y_train,w,b)
# print("Cost is ",float(total_cost))

output=predict(x_train,w,b)
print("Output is ",output)
output2=sigmoid(np.dot(x_train,w+b))
print("Sigmoid output is ",output2)
py.scatter(output,output2,c='r')
py.title("Comparison")
py.xlabel("predict")
py.ylabel("sigmoid")
py.show()

    

    
    

Read More

More operators, order, between, in, like and where clause

 

In this blog we will talk about some more operators which we can use in oracle SQL e.g not, or, and, >=, <=, <, >, =, <> etc. These are some more operators and how we can use them now we will understand this.

Before learning about these operators we should understand about where clause. As this name is showing the qualities of this clause that it uses when we want to filter the result also this we can use only with the select statement. Example to use this clause is given below

select ename from emp where empid=2;

Meaning: Meaning of this query is to find the employee name from employee table where employee if is equal to 2.

Use of the operators:

select ename, sal from emp where sal>2000;

select ename, sal from emp where sal<2000;

select ename, sal from emp where sal>=2000 or sal<=2000;

select ename, sal from emp where sal<>1000; (Not equal to operator)

This is how we can use these operators. There are also lot of the things which we can perform in oracle SQL.

Order:

We can also show the result in ascending or descending order and how we can use this is given below

select ename, sal from emp order by ename asc; (result will show alphabetically) 

select ename, sal from emp order by ename desc; (result will be show in descending order)

You can also check the result by copy the query and paste that queries to the SQL plus. 

Between, in, like:

As the name showing the meaning that between is uses to show the result between two inputs, In uses as or operator and like is uses to show the similar result for like. How we can use this examples are given below

1) select ename, sal from emp where sal between 1000 and 500; (this will show the salaries which are between 1000 and 500)

2) select ename, sal from emp where empid in(1,2,3); (this will show the ename and salaries for empid's which are 1, 2 or 3;

3) select * from emp where ename like'%R'; (this will show the results where enames end with R)

More about Like:

There are more things which are necessary to understand about the like operator so there are below

like'%R%'; (this will display the result for enames where names contain R in the center)

Read More

3) Multiple Linear Regression

 Multiple Linear Regression

Here we have multiple features such as (x1,x2,x3...) instead of one feature (x1) and after feed these features to model we can get the output (y). Before proceeding away we should understand some of the terminologies which we use for ML. let's assume a[i][j] here will the help of j we can proceed toward columns and with the help of i we can proceed toward rows. Furthermore, n is the total number features.

How Model find value for Multiple Features:

let’s assume we have 3 features x1, x2 and x3 with three weights w1, w2, w3 and one bias b and by using these values we can find the output f(x).

f(x)= ((x1*w1)+(x2*w2)+(x3*w3))+b

After getting the f(x) value we go ahead by finding the loss with the help of loss function and apply gradient decent to get the right parameters (weight and bias) for the model. This all we will understand with time. But for now we should have little bit of understanding for basic terminologies, methods and approaches.

=> We have bias b, weights [w1,w2,w3..] and features [x1,x2,x3…]

=> Model f(x)= pred_y[ i ] =w1*x1+w2*x2+…+w(n)*x(n)

=> cost function J(w[i],b)=((sigma(pred_y[i]- actual_y[i]))²)/2m

=>update weight w[i]= w[i]- learning rate * ( J(w[i], b ) * x[i]

=>update bias b= b- learning rate * (J(w[i],b)

Note: Vectorization is good approach to perform any operation as compare to use loops

Here we have Model named as f(x) and with the help of model we predict our output (y). After predicting the value (y) we will find the error using cost function called Gradient decent. Our parameters will perfectly tuned if our cost function is close to zero. With the help of cost function we will find the cost between actual_y (output) and predicted_y (output). If our error is high then we will update the weight and bias and make the cost close to 0.

Feature Scaling:

In simple word this is normalization of independent features. This helps us make gradient fast. We use normalization if either feature difference is too small or too large. Therefore, in normalization we scale the values between 0 and 1 most of the case. We can take the example of images while training model for images we normalize the images to make training fast for model. There are too much types of normalization and mostly common two of them are given below

1) Z-Score Normalization: Here data (features) will normalize in such that mean of your data will be zero and standard derivation will become 1.

               Formula:          x1= ( x1 - mean( u1 ) ) / ( standard deviation )

2) Mean Normalization:   Here we normalize the data with respect to mean

             

               Formula:           x1= ( x1 - mean( u1 ) ) / ( max - min )

Note: Now a days features engineering is one of the most useful thing in ML because with the help of it we identifies news features for model

An overview Diagram of Linear Regression:

Notes for Linear Regression:

I made very simplified version linear regression Code (Python) which is given below:

import numpy as np
import matplotlib.pyplot as plt
xFeatures=np.array([2,3])
yFeatures=np.array([330,400])
# plt.scatter(xFeatures,yFeatures,marker='x',c='r')
# plt.title("Actual Values")
# plt.xlabel("Size")
# plt.ylabel("Price")
# plt.show()
lr=0.1
m=len(xFeatures)
def calculate(x,w,b):
    y=w*x+b
    return y
def cost(x,y,w,b):
    JSumWeight=0
    m=len(x)
    for i in range(m):
        f_wb=(w*x[i])+b
        JSumWeight=JSumWeight+np.power((f_wb-y[i]),2)
    totalCostW=(1/2*m)*JSumWeight
    return totalCostW

def gDecent(x,y,w,b):
    wid=0
    bs=0
    sumWeights=0
    sumBias=0
    for i in range(len(x)):
        f_wb=(w*x[i])+b
        wid=(f_wb-y[i])*x[i]
        bs=(f_wb-y[i])
        sumWeights+=wid
        sumBias+=bs
        
    sumWeights=(1/m)*(sumWeights)
    sumBias=(1/m)*(sumBias)
    return sumWeights,sumBias
            
def resultGDecent(x,y,w,b,iterations):
    recordW=np.zeros(iterations)
    recordB=np.zeros(iterations)
    for i in range(iterations):
        errorW,errorB=gDecent(x,y,w,b)
        newW=w-(lr*errorW)
        newB=b-(lr*errorB)
        w=newW
        b=newB
        recordW[i]=w
        recordB[i]=b
    
    return recordW,recordB

bias=1
weight=0.25
w,b=resultGDecent(xFeatures,yFeatures,weight,bias,1000)
print("1st Weight is ",w)
print("1st Bias is ",b)

# w,b=resultGDecent(xFeatures,yFeatures,w,b)
# print("Last Weight is ",w)
# print("Last Bias is ",b)
# w,b=resultGDecent(xFeatures,yFeatures,w,b)
# print("Last 1 Weight is ",w)
# print("Last 1 Bias is ",b)
getCost=cost(xFeatures,yFeatures,w,b)
print("Cost is ",getCost)
# plt.scatter(w,b,marker='x',c='r')
# plt.title("Actual Values")
# plt.xlabel("Width")
# plt.ylabel("Height")
# plt.show()
print("Our real value is ",xFeatures*w[-1]+b[-1])

Output:
After updating weights and bias we will have predicted value below
Our predicted value (y) is [328.882757 400.78734259]
You can check the actual y above in the code. For features x our model is predicting these values y after updating weights and bias with the help of gradient decent.
For Now this is enough. Today we learn some of the terminologies and functions which helps us for training model and updating weights and bias. We also see the basic code for linear regression.

Read More

2) Linear Regression Model

 Linear Regression Model

This is the type of Supervised Learning Algorithms. Here we will predict a value after getting some inputs. We have some features x (if you have multiple features then x will become x1, x2, x3 and so on)as input. We will take these features and feed them to a trained model and our model will predict a value as output. In this way linear Regression Model works. Now let’s try to go in depth of model.

If we break the model as initial stage then it will be like this

Feature (x) — — — — — — — — → Model (f(x)) — — —— — — — -> Prediction (y)

Here f(x)= w*x +b, Maybe you are thinking what is this so let me explain. Here w is some weight, x is feature and b is bias. Let’s assume you bought 3 eggs (egg is feature) and price of a egg is 10 rupees (here price is weight and bias is 0(you can also take 1 as a bias). We know that weight and bias can be high or low according to the situation) Now let’s predict the price of egg as output by putting detail in the equation which we named as model

w=10; x=3; b=0

f(x)=w* x + b — — — — — — — — — -> 10 * 3 + 0 — — — — — — — — — -> 30 (this is prediction which we can say y)

Our actual price of eggs (actual-y) is 30. This means that we have 2 y’s first one is predictive y and second one is actual y. Predictive y denotes to that price which we predict by our own with the help of our model using that information which we have. And actual y denotes the actual or real price of eggs according to the market. After comparing the actual y with predictive y if both are near to each other means about the same then our model prediction is good if they are not same or near to each other then our model prediction is not good. Like other mathematic scenarios we should have some equation to measure this error between actual-y and predictive-y and here it is, to measure this error we have a method called cost function. formula is given below. 

Here n is number of training examples and Y[i] is actual value and Y^ is predicted value. We will put the values inside this function if output will be close to zero then our output is accurate otherwise we have to change the parameters (weight and bias) to make prediction accurate. Now maybe you are confusing how parameters (we are using parameter word for w and b)are changing, actually for training purpose when we are trying to train the model with the help of features x then we will check the actual-y with predictive-y at every iteration every error is high then we will change the weight and bias or update the weight and bias according to the situation. There are lot of the techniques to update the weights and bias for now you have to take the overview of the concept. We will cover the complete process how model trains itself in upcoming tutorials. At this stage you should familiar with Gradient Decent technique to update the weights and bias. Formula to update weight and bias using gradient decent is given below

Here alpha is learning rate which is also a parameter. You have to set the learning rate accurately. If Learning rate will be too small then weight and bias will update slowly toward target weight and bias (target weight and bias is that value where your model will predict the right output). If Learning rate alpha will have very large value them may be we cannot reach toward the target weight and bias. So you have to set the right value for learning rate so that our model will learn accurately and predict right value.

Example of Linear Regression using Single Neuron:

This Diagram shows the method of prediction that how model predicts the value. For today’s blog your understanding for predicting values should be clear.

Read More

1) Machine Learning

 Machine Learning

Introduction:

Machine Learning is the subset of Artificial Intelligence. Now a days this is very buzz word. If you ask from anyone what do you want to become than most of the people answer it as Machine Learning Engineer. This is not easy as sound like here you have a huge grip on Math and stats to become a good ML Engineer. Actually, This is explicit program which Help us to get label or predict label, recommend videos to anyone according to his/her searching data, predict spam email or not etc. There are various things which are using Machine Learning such as  Self Driving Cars, Speech Recognition Systems and Google Maps etc. There is another buzz word called General AI which helps us to act the AI like Humans.

Machine Learning Types:

ML is explicit program to create intelligence e.g while playing checkers ML Algorithm works better as much as ML Algorithm gets the Data. There are some types of Machine Learning Algorithms which are Given Below: 

1) Supervised Learning (It is using in real life applications)

2) Unsupervised Learning

3) Reinforcement Learning

4) Recommendation Systems

We will choose the right tool for a specific problem after identifying the problem and after getting identification we can select the right algorithm.

1) Supervised Learning:

About 99% problems are solving now a days using supervised learning. e.g

                    => email     ------------->        spam(0/1)       ---------->              spam filtering

                    => audio     ------------->        text transcripts         --->              speech recognition

                    => English  ------------->        Spanish        ------------>              Machine Translation

                    => ad, user info    ------>        click(0/1)         --------->              online advertising

                    => image, radar info  -->       position of other cars  ->              self driving cars

                    => image of phone  ---->       defect(0/1)      ---------->              visual inspection

Regression and classification are the type of supervised learning. 

  => classification  problem predict categories (it should not be number). For example you are classifying the two categories for email like spam or not spam. You can say class A is spam and class B is not spam. In classification problem your model will be able to tell the difference between these two classes after training. You should not limit the categories I choose the example of binary classification because here we are dealing with 2 classes which are spam(1) and not spam(0). There can be multiple classes in the classification problem. (Here I have one of the most important point which is your model can only understand the numbers. You have to convert everything into numbers e.g images, text). Here we also have to convert the categories into numbers like 0 and 1 if you have more than one category then convert it into 0,1,2,3 on so on. This is really a fun.

  => Regression problems predict the infinitely possible numbers. This is also one of the famous type of supervise learning. Here we can take the example of House Price Prediction System where you have to estimate the price of the house and this can be infinite possible numbers. Here you are not dealing with categories or classes here you are dealing with numbers. You will feed the training x features (features are attributes of the house like location, size, rooms, bathroom etc.) to the model with training y feature of the model price. After feeding the data your model will start finding the pattern by adjusting the parameters weight and bias(we will see the whole mechanism later in the blogs which can help you understanding the whole mechanism). Here weights and bias will adjust in this way that when features x will multiply by weight (w) and add with bias (b) you will get your predicted price (y=w*x+ b). I know this is little bit hard to understand at this stage but don’t worry we will cover every thing in detail later. We will also understand the model training mechanism, testing mechanism ans so on. I will also provide you the code which can help you for better understanding.

2) Unsupervised Learning:

Here we find something interesting with unlabeled data e.g clustering, k-mean etc.

Note: Google also use clustering in Google News to show related articles by finding similar tags or words.

Here you do not has input x with output y to train model so here we find structure in data by clustering, anomaly detection(find unusual data points), dimensionality reduction(compress big data set to small one using fewer number).

Note: Remaining two types of Algorithms we will write in upcoming tutorials

Read More

Operation of Hello World c++

 

[C++ is case sensitive language means if we will use upper case letter at the place of lower letter in the syntax then program will generate the error.] 

Here is the article related to c++ programming. Firstly we will check the simple Hello World program and then break that program into pieces and then check the every line and its use in the program

       
#include<iostream>
using namespace std;
int main()
{
cout<<"Hello World"<<endl;
return 0;

This is the program which will display the message "Hello World". Now we will break this program into pieces.

1) #include<iostream>

This is called header file in c++. We uses different kind of header files to access the different functionality of c++. This is like the key to access the treasure. In simple words if we want to access the specific file then we uses header file. In the above program I used iostream header file which is uses to take input or display the data or for basic functionalities such as to break line etc.

2) Using namespace std;

Actually the complete syntax of c++ is like

 "std::cout<<"Hello World"<<std::endl;"

 (in this way use std:: with every object used from specific header file)

 this syntax will become little bit difficult to implement therefore, we use using namespace std and this will automatically apply the std with every object of header file and in this way it will become more easy to use. 

Note:  

do you focus here at the end of the line I used a semi column(;) this we use to tell the compiler that we had end the line.

3) int main()

we can also use void at the place of int. Actually, we use main as a driver of the program like a car. Main should be compulsory in a program because without main program cannot run also every program always start from main. Moreover, this is more like a function which should be compulsory in a program. May be, in this stage you are not familiar with functions so, don't worry we will learn in our next lectures.

Note: After using int main() we will also define the scope for main by using the bracket({). If you don't know how to define the scope then answer is you can define scope like this { here you can write the program part}.

4) cout<<"Hello World"<<endl;

This is the main line to display the hello world as a output. Here I used cout and endl. cout is uses to display the data and after writing cout we need to use stream out operator(<<) and then use "Write string or message here" and then again stream out operator(<<) and then I used the endl to break the line. In this way I display the message as a output.

5) return 0;

We use return 0 when our main is of int type because int always return an integer at the end of the function. But if we will use void as the return type of main then it will return nothing at the end of the function.

In this way program works here. Now, you can try different experiments for this program to understand the behavior. 

Read More

How to develop logic?

 Logic Development

Have you heard programmer is nothing without logic? If yes then you should definitely read this article to which can help you in development of logics.

Before some practices we should understand what is logic? can you answer this question? you should answer if you don't know then don't worry I will answer you. "The ability to solve any problem". If you know better answer then you can comment below this will also increase my understand related to logics. So what ever, now I should come to point that how to develop logic?

How to develop

 Here please try to answer these questions.

1) How can you add 2 odd numbers to get even answer?

Answer:

Yes I know answer is too simple and definitely you had answered this question but I should also contribute my answer which is add 1+1 =2(odd +odd=even) . This is called logics and by solving these kinds of problems you can enhance you logics. Now try to solve question two.

2) Now try to solve by adding 3 odd numbers to get even answer?

Answer:

This is called logic now may be you are confuse here and may be your answer will be this is not possible but here I did not limit you, you can use any of the method to solve this question.

Hint: you can also use decimal numbers.

May be now you can solve this question. If not then don't worry I will tell you so the answer is 

5 + 3.9 + 1.1 =10 (odd + odd + odd= even).

Now, you had learnt the solution of another problem and do you know the meaning of it. Meaning of it, is you had develop your logic by solve these complex problems. Now you have to add this logic into programming (these are the sets of instructions which a machine follows to perform any task). If I will tell you to define that question 1 into the set of instructions. you can use any language to define instructions and at the same time I will also solve this question and answer is below

1) initialize three variables one for first number, second for second number and third for answer 

2) Now assign the first two variables with number

3) Now, add the first two variables and store the answer into third variable

4) Now you can display the answer.

I will solve this question in this way you can also use your own way to define instructions also you don't need to use the line of initialize or assigning because you don't know the meaning of it because now you are beginners. We will learn this in the next blogs.

Tips to improve Logic

1) Solve critical problems

2) try to solve same problem with different ways

3) Play logical games such as Castle Crash or Candy crash etc

4) Critical Programming practices is one of the best way to improve logics.

There are tons of others ways which can improve your logics but these are the only one's which are on my mind which I had shared.

Stay connected for more these kinds of articles.

 

Read More