module-2-linear-classifier-assignment-Done.md

import matplotlib.pyplot as plt
# %matplotlib inline
import numpy as np
import pandas as pd
from sklearn import datasets, linear_model
from sklearn.cross_validation import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
import string
from pandas import Series, DataFrame

Load Amazon dataset

products = pd.read_csv("D:\ml_data\/amazon_baby.csv")
# products = products.head(100)

Perform text cleaning

removing punctuation by pandas.str.replace

products['review_clean'] = products['review'].str.replace('[^\w\s]', '')

fill n/a values in the review column with empty strings

products = products.fillna({'review': ''})

Extract Sentiments

ignore rating = 3 that tends to a neutral sentiment

products = products[products['rating'] != 3]

define positive and negative review

products['sentiment'] = products['rating'].apply(lambda rating: +1 if rating > 3 else -1)

Split into training and test sets

(train_data,test_data) = train_test_split(products, train_size=0.7, random_state=0)

len(test_data)

50026

len(train_data)

116726

Build the word count vector for each review

use regex token pattern to keep single-letter words

vectorizer = CountVectorizer(token_pattern=r'\b\w+\b')

type(vectorizer)

sklearn.feature_extraction.text.CountVectorizer

First, learn vocabulary from the training data and assign columns to words Then convert the training data into a sparse matrix

train_matrix = vectorizer.fit_transform(train_data['review_clean'].values.astype('U'))

type(train_matrix)

scipy.sparse.csr.csr_matrix

test_matrix = vectorizer.transform(test_data['review_clean'].values.astype('U'))

Train a sentiment classifier with logistic regression

sentiment_model = linear_model.LogisticRegression()
sentiment_model.fit(train_matrix, train_data['sentiment'])

LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,
          verbose=0, warm_start=False)

sentiment_model.coef_

array([[-1.36821636e+00,  8.61907787e-04,  1.74107372e-02, ...,
         1.02410276e-02,  2.80385130e-03, -7.17964970e-05]])

type(sentiment_model.coef_)

numpy.ndarray

Making predictions with logistic regression

sample_test_data = test_data[10:13]
print(sample_test_data)

                                                    name  \
27972             Bumkins Waterproof Superbib, Blue Fizz   
123316                         Safety 1st High Door Lock   
60874   DadGear Courier Diaper Bag - Orange Retro Stripe   

                                                   review  rating  \
27972   I love these bibs!  We have about 8 of them in...       5   
123316  My 4 year old gets up earlier than me, this me...       5   
60874   love the bag, especially since it's over the s...       4   

                                             review_clean  sentiment  
27972   I love these bibs  We have about 8 of them in ...          1  
123316  My 4 year old gets up earlier than me this mea...          1  
60874   love the bag especially since its over the sho...          1  

sample_test_data['review'].index

Int64Index([27972, 123316, 60874], dtype='int64')

# sample_test_data['review'][113598]
# sample_test_data['review'][8]

len(sample_test_data['review'])

3

for s in sample_test_data['review']:
    print(s)
    print('___________________________________')

I love these bibs!  We have about 8 of them in different patterns!  They was well!  Even with bleach!We've been using them for about 4 years!
___________________________________
My 4 year old gets up earlier than me, this means if he can be quiet he will sneak downstairs and steal sugar.  I don't usually keep treats (cookies, candies, etc) in the house so he will steal actual sugar.  We tried different disciplines but nothing worked.  He has told us "I don't want to steal the sugar but my belly tells me to."  We tried all sorts of locks, we needed something to go on the door to the downstairs and not his room b/c he needs to be able to leave to use the bathroom.  Now, I can lock the door at night and know that he won't be able to sneak down in the morning and steal sugar, or use knives, or the stove, you know, the typical stuff that 4 year olds try to do when they wake up in the morning (oh, that's not typical you say...hmmmm).  We had tried all the typical door knob locks, and some no drill locks (we were trying not to put holes in as we rent), none of them work, he can defeat them all; they tell me he's gifted...yay for me?  So even though this one requires some drilling it is really a snap to install and as long as you mount it high enough (and there are no available chairs, ladders, precariously stacked boxes available) your child shouldn't be able to get through the door.I should also note, it is easy to open from both sides of the door when locked.  When we first installed it was a little tricky to slide it from the on to off position (red button means it will not automatically lock when you shut the door, green means it will) but it seems it just needed some use to loosen up since it is easy to do now.  I also like the finger jam guard, the 4 year old has a tendency to slam doors, which often has disastrous results when his 2 year old brother acts like his shadow.
___________________________________
love the bag, especially since it's over the shoulder.  i've seen too many peeps with regular shoulder bags sliding down their arm while trying to hold a car seat or baby.  it has a lot of compartments to keep stuff organized and still have enough space in the big pocket for formula and diapers.i have to stop using the wipe holder tho.  it has always been so hard to open, i took skin off my finger today trying to open it.  and it's very hard to fill the wipe holder too.  i haven't found a way to actually remove the holder, so i'll probably just take the back off (so there's not a giant hole in my bag) and use a regular wipe holder inside the bag.
___________________________________

sample_test_matrix = vectorizer.transform(sample_test_data['review_clean'].values.astype('U'))
scores_1 = sentiment_model.decision_function(sample_test_matrix)
print(scores_1)

[ 5.67446022 15.91839528  3.78700706]

sample_test_matrix.shape

(3, 110870)

scores_2 = sentiment_model.predict(sample_test_matrix)
print(scores_2)

[1 1 1]

Probability Predictions

import math

math.e

2.718281828459045

def probability(score):
    return 1/(1 + math.e**(-score))

for i in scores_1:
    print(probability(i))

0.9965792205996046
0.9999998778963308
0.977838913549031

Find the most positive (and negative) review

We now turn to examining the full test dataset, test_data, and use sklearn.linear_model.LogisticRegression to form predictions on all of the test data points.

proba = sentiment_model.predict_proba(sample_test_matrix)

proba

array([[3.42077940e-03, 9.96579221e-01],
       [1.22103669e-07, 9.99999878e-01],
       [2.21610865e-02, 9.77838914e-01]])

Using the sentiment_model, find the 20 reviews in the entire test_data with the highest probability of being classified as a positive review. We refer to these as the "most positive reviews."

proba_all = sentiment_model.predict_proba(test_matrix)
# test_data['probability'] = sentiment_model.predict(test_data, output_type='probability')

# len(proba_all)
type(proba_all)

numpy.ndarray

proba_all

array([[1.64768828e-02, 9.83523117e-01],
       [2.13707041e-01, 7.86292959e-01],
       [1.10050642e-01, 8.89949358e-01],
       ...,
       [9.10265794e-01, 8.97342059e-02],
       [9.11668969e-02, 9.08833103e-01],
       [2.38539982e-06, 9.99997615e-01]])

proba_all_positive = proba_all[:,1]

len(proba_all_positive)

50026

type(proba_all_positive)

numpy.ndarray

type(test_data)

pandas.core.frame.DataFrame

test_data.loc[:,'probability'] = proba_all_positive

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\indexing.py:357: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self.obj[key] = _infer_fill_value(value)
C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\indexing.py:537: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self.obj[item] = s

np.sort(proba_all,axis=0)

array([[0.00000000e+00, 5.28769314e-19],
       [0.00000000e+00, 8.76249710e-16],
       [0.00000000e+00, 4.69016318e-15],
       ...,
       [1.00000000e+00, 1.00000000e+00],
       [1.00000000e+00, 1.00000000e+00],
       [1.00000000e+00, 1.00000000e+00]])

test_data

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	name	review	rating	review_clean	sentiment	probability
103733	Luvable Friends Applique Side Closure Bib, Pink	Got this for a friend for her baby shower. Lov...	5	Got this for a friend for her baby shower Love...	1	9.835231e-01
157975	Infant Optics DXR5 Wall Socket Power Adapter (...	Ok, so obviously there isn't much to say about...	5	Ok so obviously there isnt much to say about a...	1	7.862930e-01
127693	NUK Active Silicone Spout Learning Cup, Ladybu...	We have a couple versions of Tommie Tippie and...	5	We have a couple versions of Tommie Tippie and...	1	8.899494e-01
113718	Luvable Friends 12 Pack Washcloths, Blue	Ordered washcloths pictured but others were se...	1	Ordered washcloths pictured but others were se...	-1	3.017619e-01
20416	Infantino Cart Safari w/Microban	perfect for what its supposed to be. my 9 mont...	5	perfect for what its supposed to be my 9 month...	1	9.995342e-01
62236	Sesame Street Bath Tub Faucet Cover - Elmo	Thought this would be great as my son loves El...	1	Thought this would be great as my son loves El...	-1	3.682715e-01
93741	The First Years Ignite Stroller	We purchased this for a trip to Disney. My 4 y...	5	We purchased this for a trip to Disney My 4 ye...	1	9.991669e-01
182819	Bath Letters And Numbers With Bath Toys Organi...	Exactly what I was looking for. Foam letters f...	5	Exactly what I was looking for Foam letters fo...	1	9.955896e-01
108942	Britax B-Ready Stroller, Black	We got this stroller during Britax's ride even...	5	We got this stroller during Britaxs ride event...	1	1.000000e+00
63171	Ju-Ju-Be Paci Pod Pacifier Holder, Lilac Lace	The kids at my house have moved on past binkie...	5	The kids at my house have moved on past binkie...	1	8.832268e-01
27972	Bumkins Waterproof Superbib, Blue Fizz	I love these bibs! We have about 8 of them in...	5	I love these bibs We have about 8 of them in ...	1	9.965792e-01
123316	Safety 1st High Door Lock	My 4 year old gets up earlier than me, this me...	5	My 4 year old gets up earlier than me this mea...	1	9.999999e-01
60874	DadGear Courier Diaper Bag - Orange Retro Stripe	love the bag, especially since it's over the s...	4	love the bag especially since its over the sho...	1	9.778389e-01
89381	Prince Lionheart weePOD, Green	The splash guard on this seat was the major se...	5	The splash guard on this seat was the major se...	1	9.897353e-01
146248	Prince Lionheart Seat Neat, Brown/Tan	Have used it for about 3 months now and it's a...	5	Have used it for about 3 months now and its a ...	1	9.998925e-01
167897	Wrapsody Breeze Baby Carrier, Freya, Medium/Large	This wrap is very thin and lightweight, great ...	5	This wrap is very thin and lightweight great f...	1	9.460426e-01
129689	Noah And Friends High Chair Cover		5	NaN	1	8.650732e-01
54150	Medela Pump in Style Advanced Backpack	I got this pump as I went back to work with my...	5	I got this pump as I went back to work with my...	1	8.229963e-01
8352	Baby Bjorn Toilet Trainer	Before I had purchased this toilet trainer, I ...	5	Before I had purchased this toilet trainer I w...	1	9.997415e-01
55700	HABA Blossom Butterfly Mobile	My son is 6 weeks old, and we bought this when...	5	My son is 6 weeks old and we bought this when ...	1	9.980966e-01
83479	Dreambaby Super Toy Hammock and Toy Chain	Unfortunately I like in a cement high rise. I ...	5	Unfortunately I like in a cement high rise I c...	1	6.319987e-01
93618	Sunshine Kids Seat Belt Pillow, Grey	I purchased this after my husband had bypass s...	5	I purchased this after my husband had bypass s...	1	9.989821e-01
130762	Dreambaby Sliding Door and Window Locks	The product is expensive and we tried to use i...	2	The product is expensive and we tried to use i...	-1	5.309546e-02
73991	Delta Canton 4-in-1 Convertible Crib, Dark Cherry	Really nice looking, easy to put together.Look...	5	Really nice looking easy to put togetherLooks ...	1	9.942311e-01
174021	Thermos FUNtainer Monsters University Food Jar...	The thermos is great for keeping food warm. Th...	4	The thermos is great for keeping food warm The...	1	9.270045e-01
17830	Prince Lionheart Multi-Purpose Toy Hammock	When I first saw the box I was discouraged, I ...	5	When I first saw the box I was discouraged I t...	1	9.919657e-01
62700	Vulli Chan Pie Gnon Natural Rubber Teether - B...	I got this instead of the Sophie Giraffe for m...	4	I got this instead of the Sophie Giraffe for m...	1	5.626660e-01
33106	Lamaze Play & Grow Mortimer the Moose Take...	This is my son's favorite toy. He is 6 months ...	5	This is my sons favorite toy He is 6 months an...	1	9.957130e-01
156734	THE TWIN Z PILLOW - BLUE COLOR COVER The only ...	The few timew i hace used it, my back ends up ...	2	The few timew i hace used it my back ends up a...	-1	3.840617e-01
67422	Fisher-Price Cheer for Me Potty	I bought this for our third son. When I read t...	5	I bought this for our third son When I read th...	1	1.000000e+00
...	...	...	...	...	...	...
29152	Elegant Baby 8 Piece Bath Squirties Gift Set i...	I added these to my daughters wish list thinki...	5	I added these to my daughters wish list thinki...	1	9.992346e-01
145868	The First Years Deluxe Reclining Feeding Seat,...	We use this for travelling or at family's hous...	4	We use this for travelling or at familys house...	1	8.718506e-01
133924	C.R. Gibson Keepsake Memory Book of Baby's Fir...	What a great idea to have such a special keeps...	4	What a great idea to have such a special keeps...	1	8.313210e-01
144259	Baby Book in a Box Memory Keepsake Organizer	Wish I had discovered this when my children we...	5	Wish I had discovered this when my children we...	1	9.996570e-01
155251	VTech Communications Safe & Sound Video Ca...	I love having a second camera. It works great...	5	I love having a second camera It works great ...	1	9.991212e-01
49516	Mary Meyer Christening Lamb Blanket - 14 Inches	The lamb was exactly as pictured and arrived o...	4	The lamb was exactly as pictured and arrived o...	1	6.219870e-02
139738	Fisher-Price Calming Waters Vibration Bathing Tub	This is perfect for my newborn! She loves bath...	5	This is perfect for my newborn She loves bath ...	1	9.999232e-01
155993	Similac SimplySmart Bottle, 4 Ounce	I bought these bottles for my Grand daughter. ...	5	I bought these bottles for my Grand daughter T...	1	9.979781e-01
28007	Badger Basket Three Basket Set, Pink	I got a pinkmone and it was barely pink. It as...	4	I got a pinkmone and it was barely pink It as ...	1	3.436229e-01
124413	Summer Infant Lil' Luxuries Whirlpool, Bubblin...	There's not much I could say about this tub th...	1	Theres not much I could say about this tub tha...	-1	4.901535e-04
114981	Fisher-Price Sing with Me Step Stool	Bought this for my 3 year old grandson to enco...	5	Bought this for my 3 year old grandson to enco...	1	9.977971e-01
115743	Kinderglo Portable Fun and Safe Rechargeable N...	My daughter was so excited to get this night l...	1	My daughter was so excited to get this night l...	-1	1.153964e-02
109348	WubbaNub Lamb	This most adorable lamb has made giving a paci...	5	This most adorable lamb has made giving a paci...	1	9.999833e-01
73490	Regalo Easy Open 50 Inch Super Wide Walk Thru ...	Returned item. With a 10 month old who likes t...	1	Returned item With a 10 month old who likes to...	-1	2.238597e-01
10461	Regalo Easy Diner Portable Hook-On High Chair	Bought two as baby shower gifts - I used a cha...	5	Bought two as baby shower gifts I used a chai...	1	9.870586e-01
120390	Munchkin Extending Extra Tall and Wide Metal G...	Assembly wasn't too terrible - took about an h...	2	Assembly wasnt too terrible took about an hou...	-1	2.389540e-10
56823	Bunnies by the Bay Wee Plush, Ittybit	I bought this bunny for my 4 month old grand d...	5	I bought this bunny for my 4 month old grand d...	1	9.558778e-01
110189	eWonderWorld 24" X 24" X~9/16" ...	Perfect for a little one who is learning to cr...	4	Perfect for a little one who is learning to cr...	1	9.975703e-01
151692	WubbaNub Infant Plush Pacifier - Limited Editi...	My son has always loved these. So when I seen ...	5	My son has always loved these So when I seen t...	1	9.487158e-01
134663	Infant Optics DXR-5 2.4 GHz Digital Video Baby...	I purchased this monitor in November 2012, jus...	4	I purchased this monitor in November 2012 just...	1	1.000000e+00
14881	Magic Moments Learning Seat	We purchased this bouncer for our 2.5 month ol...	5	We purchased this bouncer for our 25 month old...	1	9.995947e-01
106787	Wubbanub Infant Pacifier ~ Giraffe & Lion	My boys love Wubba Nubba. I'm almost sad to s...	5	My boys love Wubba Nubba Im almost sad to say...	1	8.969469e-01
9918	Cosco Alpha Omega Elite Convertible Car Seat	This car seat does have a bad smell. I took a...	4	This car seat does have a bad smell I took al...	1	9.993759e-01
79920	Simple Wishes Hands-Free Breastpump Bra, Pink,...	I ordered this product after reading that the ...	5	I ordered this product after reading that the ...	1	9.647574e-01
31318	Prince Lionheart Wheely Bug, Ladybug, Large	This is the best ride-on toy I've seen. The w...	5	This is the best rideon toy Ive seen The whee...	1	9.950470e-01
54722	Graco Nautilus 3-in-1 Car Seat, Matrix	I bought this car seat for my grand daughter, ...	2	I bought this car seat for my grand daughter w...	-1	9.279842e-01
106191	Lamaze High-Contrast Flip-Flop Bug Rattle	One of her favorite toys. Excellent soft rattl...	5	One of her favorite toys Excellent soft rattle...	1	9.997943e-01
79222	Britax Chaperone Stroller, Red Mill	way too expensive i really wanted this travel ...	1	way too expensive i really wanted this travel ...	-1	8.973421e-02
93173	Contours Options Tandem II Stroller, Tangerine	The only reason I am writing this review is be...	1	The only reason I am writing this review is be...	-1	9.088331e-01
86465	Fisher-Price Newborn Rock 'n Play Sleeper, Yellow	Update: My son is now 13 months old. He slept...	5	Update My son is now 13 months old He slept i...	1	9.999976e-01

50026 rows × 6 columns

type(test_data)

pandas.core.frame.DataFrame

test_data.sort_values(by='probability', ascending=False, na_position='first')

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	name	review	rating	review_clean	sentiment	probability
147975	Baby Jogger City Mini GT Single Stroller, Shad...	Let me start by saying that I have gone throug...	5	Let me start by saying that I have gone throug...	1	1.000000e+00
167047	Inglesina 2013 Trip Stroller, Lampone Purple	I did many hours of research reading reviews a...	5	I did many hours of research reading reviews a...	1	1.000000e+00
134265	UPPAbaby Cruz Stroller, Denny	We bought this stroller after selling our belo...	5	We bought this stroller after selling our belo...	1	1.000000e+00
106455	Quinny Senzz 2011 Fashion Stroller, Star	I am very pleased overall with the Quinny Senz...	4	I am very pleased overall with the Quinny Senz...	1	1.000000e+00
171041	CTA Digital 2-in-1 iPotty with Activity Seat f...	I'll just say in advance that the haters can j...	5	Ill just say in advance that the haters can ju...	1	1.000000e+00
76549	Britax Advocate 65 CS Click & Safe Convert...	The Britax Advocate CS appears similar to the ...	4	The Britax Advocate CS appears similar to the ...	1	1.000000e+00
59320	Evenflo Aura Select Travel System - Caroline	I'm about to be a first-time mom, so I spent w...	5	Im about to be a firsttime mom so I spent week...	1	1.000000e+00
123632	Zooper 2011 Waltz Standard Stroller, Flax Brown	I did a TON of research before I purchased thi...	5	I did a TON of research before I purchased thi...	1	1.000000e+00
41763	Kolcraft Contours Lite Stroller Plus with iPod...	After considering several lightweight stroller...	4	After considering several lightweight stroller...	1	1.000000e+00
48158	Phil & Ted's "2008 Version 2" Sp...	We're keeping this stroller! After much resear...	4	Were keeping this stroller After much research...	1	1.000000e+00
88680	ERGObaby Original Baby Carrier, Galaxy Grey	After reading many online reviews, asking othe...	5	After reading many online reviews asking other...	1	1.000000e+00
21557	Joovy Caboose Stand On Tandem Stroller, Black	Ok, I read all the reviews already posted here...	5	Ok I read all the reviews already posted here ...	1	1.000000e+00
93690	The First Years Ignite Stroller	The last thing we wanted was to purchase more ...	5	The last thing we wanted was to purchase more ...	1	1.000000e+00
108803	Chicco Keyfit 22 Pound Infant Car Seat And Bas...	I bought this right before the KeyFit 30 came ...	5	I bought this right before the KeyFit 30 came ...	1	1.000000e+00
88659	ERGObaby Original Baby Carrier, Galaxy Grey	We purchased this carrier after a recommendati...	5	We purchased this carrier after a recommendati...	1	1.000000e+00
166929	Britax Pavilion 70-G3 Convertible Car Seat Sea...	We LOVE this seat! As parents to 8 children ra...	5	We LOVE this seat As parents to 8 children ran...	1	1.000000e+00
103186	Thirsties Duo Wrap Snap, Ocean Blue, Size One ...	I am reviewing these covers because reading re...	5	I am reviewing these covers because reading re...	1	1.000000e+00
152013	UPPAbaby Vista Stroller, Denny	I researched strollers for months and months b...	5	I researched strollers for months and months b...	1	1.000000e+00
158209	Ubbi Cloth Diaper Pail Liner	(updated 3.22.13) After extensive research, tr...	5	updated 32213 After extensive research trial a...	1	1.000000e+00
50735	Joovy Zoom 360 Swivel Wheel Jogging Stroller, ...	The joovy zoom 360 was the perfect solution fo...	5	The joovy zoom 360 was the perfect solution fo...	1	1.000000e+00
140780	Diono RadianR100 Convertible Car Seat, Dune	i bought this when the seat was owned by Sunsh...	5	i bought this when the seat was owned by Sunsh...	1	1.000000e+00
108943	Britax B-Ready Stroller, Black	Some differences with Uppababy Vs. Britax B-Re...	4	Some differences with Uppababy Vs Britax BRead...	1	1.000000e+00
111746	Baby Jogger 2011 City Mini Double Stroller, Bl...	Before purchasing this stroller, I read severa...	5	Before purchasing this stroller I read several...	1	1.000000e+00
135152	Maxi-Cosi Pria 70 with Tiny Fit Convertible Ca...	We've been using Britax for our boy (now 14 mo...	5	Weve been using Britax for our boy now 14 mont...	1	1.000000e+00
129722	Bumbleride 2011 Flite Lightweight Compact Trav...	This is a review of the 2012 Bumbleride Flite ...	5	This is a review of the 2012 Bumbleride Flite ...	1	1.000000e+00
100166	Infantino Wrap and Tie Baby Carrier, Black Blu...	I bought this carrier when my daughter was abo...	5	I bought this carrier when my daughter was abo...	1	1.000000e+00
129212	Baby Jogger 2011 City Select Stroller in Ameth...	I have the Baby Jogger City Select with Second...	5	I have the Baby Jogger City Select with Second...	1	1.000000e+00
14008	Stork Craft Beatrice Combo Tower Chest, White	I bought the tower despite the bad reviews and...	5	I bought the tower despite the bad reviews and...	1	1.000000e+00
162687	Joovy Caboose Too Rear Seat, Greenie	We are thrilled with this rear seat. This litt...	5	We are thrilled with this rear seat This littl...	1	1.000000e+00
166827	Britax Boulevard 70-G3 Convertible Car Seat Se...	We just purchased this seat for our eight mont...	5	We just purchased this seat for our eight mont...	1	1.000000e+00
...	...	...	...	...	...	...
4835	JJ Cole Premaxx Sling Carrier - New Edition Re...	I bought this for my daughter while I was stil...	1	I bought this for my daughter while I was stil...	-1	1.595787e-09
3746	Playtex Diaper Genie - First Refill Included	Prior to parenthood, I had heard several paren...	1	Prior to parenthood I had heard several parent...	-1	1.505042e-09
13712	Badger Basket Elegance Round Baby Bassinet, Wh...	I registered for this item & it was ordered fo...	1	I registered for this item it was ordered for...	-1	1.412743e-09
5033	Playtex 3 Pack BPA Free VentAire Wide Bottles,...	Initially, I thought these angled bottles make...	1	Initially I thought these angled bottles make ...	-1	1.154985e-09
7075	Peace of Mind Two 900 Mhz Baby Receivers, Monitor	If we only knew when we registered how terribl...	1	If we only knew when we registered how terribl...	-1	1.019131e-09
115108	Blueberry Deluxe Diaper, Cow	I purchased the Blueberry One Sized Bamboo Del...	2	I purchased the Blueberry One Sized Bamboo Del...	-1	9.164181e-10
70137	Lulu Ladybug Rocker by Rockabye	Here's my letter I sent to Rockabye:I purchase...	1	Heres my letter I sent to RockabyeI purchased ...	-1	4.367577e-10
72262	Baby Trend High Chair, Chickadee	We recently moved from Okinawa back to America...	2	We recently moved from Okinawa back to America...	-1	4.106157e-10
124145	Philips AVENT BPA Free Twin Electric Breast Pump	UPDATED REVIEW:So, after 2 month of use (once ...	1	UPDATED REVIEWSo after 2 month of use once a d...	-1	3.359271e-10
41581	Newborn Baby Pea in The Pod Halloween Costume,...	Looks really cute, however, the cloth smells f...	1	Looks really cute however the cloth smells fun...	-1	3.141029e-10
67615	Fisher-Price Butterfly Garden Cradle 'n Swing ...	My 2nd daughter is 10 weeks old and a little c...	1	My 2nd daughter is 10 weeks old and a little c...	-1	2.780037e-10
120390	Munchkin Extending Extra Tall and Wide Metal G...	Assembly wasn't too terrible - took about an h...	2	Assembly wasnt too terrible took about an hou...	-1	2.389540e-10
107148	Tadpoles 36 Sq Ft ABC Floor Mat, Pink/Brown	I have read the reviews after I bought these (...	4	I have read the reviews after I bought these r...	1	1.938740e-10
54418	Fisher-Price Zen Collection Gliding Bassinet	THIS BASSINET IS OVERPRICED AND RIDICULOUS. I...	2	THIS BASSINET IS OVERPRICED AND RIDICULOUS If...	-1	1.801620e-10
75995	Peg-Perego Tatamia High Chair, White Latte	Edited to Add 6/4/2010: Just wanted to add th...	1	Edited to Add 642010 Just wanted to add that ...	-1	5.132764e-11
31741	Regalo My Cot Portable Bed, Royal Blue	If I could give this product zero stars I woul...	1	If I could give this product zero stars I woul...	-1	4.603205e-11
78811	Tike Tech Single City X4 Swivel Stroller, Paci...	I initially thought this was a very sturdy, lo...	2	I initially thought this was a very sturdy lon...	-1	3.269938e-11
73851	Stork Craft Rochester Stages Crib with Drawer	I ordered this crib because I really liked the...	2	I ordered this crib because I really liked the...	-1	3.203031e-11
145109	Withings Smart Baby Monitor, White	Ok, the good. During the day, the quality of ...	1	Ok the good During the day the quality of the...	-1	2.419640e-11
179566	Summer Infant Dual Coverage Digital Color Vide...	This was possibly the most disappointing elect...	2	This was possibly the most disappointing elect...	-1	9.570229e-12
38270	Nuby No Spill Flip-it Cup, 12 Ounce, Colors Ma...	I had a frustrating experience with these two ...	4	I had a frustrating experience with these two ...	1	5.553832e-12
20331	Cabinet Flex-Lock (2 pack) from Safety First	I am a mother of 4 children, ages 6-18 months....	5	I am a mother of 4 children ages 618 months I...	1	4.182831e-13
57234	Dream On Me Bassinet, Blue	My husband and I are VERY disappointed and sho...	1	My husband and I are VERY disappointed and sho...	-1	2.279526e-13
48694	Adiri BPA Free Natural Nurser Ultimate Bottle ...	I will try to write an objective review of the...	2	I will try to write an objective review of the...	-1	1.520506e-13
92570	Munchkin Arm and Hammer Diaper Pail, White	I would recommend in the strongest possible wa...	1	I would recommend in the strongest possible wa...	-1	7.840347e-14
99594	Valco Baby Tri-mode Twin Stroller EX- Hot Choc...	I give one star to the dimension: 1. being 29 ...	1	I give one star to the dimension 1 being 29 in...	-1	6.592510e-14
131738	Kids Line Cascade Bow Diaper Bag, Black	I purchased this in the black color. For some...	2	I purchased this in the black color For some ...	-1	1.286862e-14
10370	Wimmer-Ferguson Infant Stim-Mobile	This product should be in the hall of fame sol...	1	This product should be in the hall of fame sol...	-1	4.690163e-15
89902	Peg-Perego Aria Twin Stroller, Java	I am so incredibly disappointed with the strol...	1	I am so incredibly disappointed with the strol...	-1	8.762497e-16
10180	Arms Reach Co-Sleeper brand Mini Co-Sleeper Ba...	Please see my email to the company:Hello,I am ...	1	Please see my email to the companyHelloI am wr...	-1	5.287693e-19

50026 rows × 6 columns

test_data.sort_values(by='probability', ascending=True, na_position='first')

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	name	review	rating	review_clean	sentiment	probability
10180	Arms Reach Co-Sleeper brand Mini Co-Sleeper Ba...	Please see my email to the company:Hello,I am ...	1	Please see my email to the companyHelloI am wr...	-1	5.287693e-19
89902	Peg-Perego Aria Twin Stroller, Java	I am so incredibly disappointed with the strol...	1	I am so incredibly disappointed with the strol...	-1	8.762497e-16
10370	Wimmer-Ferguson Infant Stim-Mobile	This product should be in the hall of fame sol...	1	This product should be in the hall of fame sol...	-1	4.690163e-15
131738	Kids Line Cascade Bow Diaper Bag, Black	I purchased this in the black color. For some...	2	I purchased this in the black color For some ...	-1	1.286862e-14
99594	Valco Baby Tri-mode Twin Stroller EX- Hot Choc...	I give one star to the dimension: 1. being 29 ...	1	I give one star to the dimension 1 being 29 in...	-1	6.592510e-14
92570	Munchkin Arm and Hammer Diaper Pail, White	I would recommend in the strongest possible wa...	1	I would recommend in the strongest possible wa...	-1	7.840347e-14
48694	Adiri BPA Free Natural Nurser Ultimate Bottle ...	I will try to write an objective review of the...	2	I will try to write an objective review of the...	-1	1.520506e-13
57234	Dream On Me Bassinet, Blue	My husband and I are VERY disappointed and sho...	1	My husband and I are VERY disappointed and sho...	-1	2.279526e-13
20331	Cabinet Flex-Lock (2 pack) from Safety First	I am a mother of 4 children, ages 6-18 months....	5	I am a mother of 4 children ages 618 months I...	1	4.182831e-13
38270	Nuby No Spill Flip-it Cup, 12 Ounce, Colors Ma...	I had a frustrating experience with these two ...	4	I had a frustrating experience with these two ...	1	5.553832e-12
179566	Summer Infant Dual Coverage Digital Color Vide...	This was possibly the most disappointing elect...	2	This was possibly the most disappointing elect...	-1	9.570229e-12
145109	Withings Smart Baby Monitor, White	Ok, the good. During the day, the quality of ...	1	Ok the good During the day the quality of the...	-1	2.419640e-11
73851	Stork Craft Rochester Stages Crib with Drawer	I ordered this crib because I really liked the...	2	I ordered this crib because I really liked the...	-1	3.203031e-11
78811	Tike Tech Single City X4 Swivel Stroller, Paci...	I initially thought this was a very sturdy, lo...	2	I initially thought this was a very sturdy lon...	-1	3.269938e-11
31741	Regalo My Cot Portable Bed, Royal Blue	If I could give this product zero stars I woul...	1	If I could give this product zero stars I woul...	-1	4.603205e-11
75995	Peg-Perego Tatamia High Chair, White Latte	Edited to Add 6/4/2010: Just wanted to add th...	1	Edited to Add 642010 Just wanted to add that ...	-1	5.132764e-11
54418	Fisher-Price Zen Collection Gliding Bassinet	THIS BASSINET IS OVERPRICED AND RIDICULOUS. I...	2	THIS BASSINET IS OVERPRICED AND RIDICULOUS If...	-1	1.801620e-10
107148	Tadpoles 36 Sq Ft ABC Floor Mat, Pink/Brown	I have read the reviews after I bought these (...	4	I have read the reviews after I bought these r...	1	1.938740e-10
120390	Munchkin Extending Extra Tall and Wide Metal G...	Assembly wasn't too terrible - took about an h...	2	Assembly wasnt too terrible took about an hou...	-1	2.389540e-10
67615	Fisher-Price Butterfly Garden Cradle 'n Swing ...	My 2nd daughter is 10 weeks old and a little c...	1	My 2nd daughter is 10 weeks old and a little c...	-1	2.780037e-10
41581	Newborn Baby Pea in The Pod Halloween Costume,...	Looks really cute, however, the cloth smells f...	1	Looks really cute however the cloth smells fun...	-1	3.141029e-10
124145	Philips AVENT BPA Free Twin Electric Breast Pump	UPDATED REVIEW:So, after 2 month of use (once ...	1	UPDATED REVIEWSo after 2 month of use once a d...	-1	3.359271e-10
72262	Baby Trend High Chair, Chickadee	We recently moved from Okinawa back to America...	2	We recently moved from Okinawa back to America...	-1	4.106157e-10
70137	Lulu Ladybug Rocker by Rockabye	Here's my letter I sent to Rockabye:I purchase...	1	Heres my letter I sent to RockabyeI purchased ...	-1	4.367577e-10
115108	Blueberry Deluxe Diaper, Cow	I purchased the Blueberry One Sized Bamboo Del...	2	I purchased the Blueberry One Sized Bamboo Del...	-1	9.164181e-10
7075	Peace of Mind Two 900 Mhz Baby Receivers, Monitor	If we only knew when we registered how terribl...	1	If we only knew when we registered how terribl...	-1	1.019131e-09
5033	Playtex 3 Pack BPA Free VentAire Wide Bottles,...	Initially, I thought these angled bottles make...	1	Initially I thought these angled bottles make ...	-1	1.154985e-09
13712	Badger Basket Elegance Round Baby Bassinet, Wh...	I registered for this item & it was ordered fo...	1	I registered for this item it was ordered for...	-1	1.412743e-09
3746	Playtex Diaper Genie - First Refill Included	Prior to parenthood, I had heard several paren...	1	Prior to parenthood I had heard several parent...	-1	1.505042e-09
4835	JJ Cole Premaxx Sling Carrier - New Edition Re...	I bought this for my daughter while I was stil...	1	I bought this for my daughter while I was stil...	-1	1.595787e-09
...	...	...	...	...	...	...
74899	Graco Blossom Highchair, Townsend	We love this highchair. We have a 4 year old ...	5	We love this highchair We have a 4 year old a...	1	1.000000e+00
162687	Joovy Caboose Too Rear Seat, Greenie	We are thrilled with this rear seat. This litt...	5	We are thrilled with this rear seat This littl...	1	1.000000e+00
14008	Stork Craft Beatrice Combo Tower Chest, White	I bought the tower despite the bad reviews and...	5	I bought the tower despite the bad reviews and...	1	1.000000e+00
135152	Maxi-Cosi Pria 70 with Tiny Fit Convertible Ca...	We've been using Britax for our boy (now 14 mo...	5	Weve been using Britax for our boy now 14 mont...	1	1.000000e+00
108943	Britax B-Ready Stroller, Black	Some differences with Uppababy Vs. Britax B-Re...	4	Some differences with Uppababy Vs Britax BRead...	1	1.000000e+00
108803	Chicco Keyfit 22 Pound Infant Car Seat And Bas...	I bought this right before the KeyFit 30 came ...	5	I bought this right before the KeyFit 30 came ...	1	1.000000e+00
21557	Joovy Caboose Stand On Tandem Stroller, Black	Ok, I read all the reviews already posted here...	5	Ok I read all the reviews already posted here ...	1	1.000000e+00
152013	UPPAbaby Vista Stroller, Denny	I researched strollers for months and months b...	5	I researched strollers for months and months b...	1	1.000000e+00
111746	Baby Jogger 2011 City Mini Double Stroller, Bl...	Before purchasing this stroller, I read severa...	5	Before purchasing this stroller I read several...	1	1.000000e+00
140780	Diono RadianR100 Convertible Car Seat, Dune	i bought this when the seat was owned by Sunsh...	5	i bought this when the seat was owned by Sunsh...	1	1.000000e+00
59320	Evenflo Aura Select Travel System - Caroline	I'm about to be a first-time mom, so I spent w...	5	Im about to be a firsttime mom so I spent week...	1	1.000000e+00
171041	CTA Digital 2-in-1 iPotty with Activity Seat f...	I'll just say in advance that the haters can j...	5	Ill just say in advance that the haters can ju...	1	1.000000e+00
106455	Quinny Senzz 2011 Fashion Stroller, Star	I am very pleased overall with the Quinny Senz...	4	I am very pleased overall with the Quinny Senz...	1	1.000000e+00
158209	Ubbi Cloth Diaper Pail Liner	(updated 3.22.13) After extensive research, tr...	5	updated 32213 After extensive research trial a...	1	1.000000e+00
88659	ERGObaby Original Baby Carrier, Galaxy Grey	We purchased this carrier after a recommendati...	5	We purchased this carrier after a recommendati...	1	1.000000e+00
88680	ERGObaby Original Baby Carrier, Galaxy Grey	After reading many online reviews, asking othe...	5	After reading many online reviews asking other...	1	1.000000e+00
123632	Zooper 2011 Waltz Standard Stroller, Flax Brown	I did a TON of research before I purchased thi...	5	I did a TON of research before I purchased thi...	1	1.000000e+00
93690	The First Years Ignite Stroller	The last thing we wanted was to purchase more ...	5	The last thing we wanted was to purchase more ...	1	1.000000e+00
134265	UPPAbaby Cruz Stroller, Denny	We bought this stroller after selling our belo...	5	We bought this stroller after selling our belo...	1	1.000000e+00
41763	Kolcraft Contours Lite Stroller Plus with iPod...	After considering several lightweight stroller...	4	After considering several lightweight stroller...	1	1.000000e+00
147975	Baby Jogger City Mini GT Single Stroller, Shad...	Let me start by saying that I have gone throug...	5	Let me start by saying that I have gone throug...	1	1.000000e+00
100166	Infantino Wrap and Tie Baby Carrier, Black Blu...	I bought this carrier when my daughter was abo...	5	I bought this carrier when my daughter was abo...	1	1.000000e+00
167047	Inglesina 2013 Trip Stroller, Lampone Purple	I did many hours of research reading reviews a...	5	I did many hours of research reading reviews a...	1	1.000000e+00
48158	Phil & Ted's "2008 Version 2" Sp...	We're keeping this stroller! After much resear...	4	Were keeping this stroller After much research...	1	1.000000e+00
129212	Baby Jogger 2011 City Select Stroller in Ameth...	I have the Baby Jogger City Select with Second...	5	I have the Baby Jogger City Select with Second...	1	1.000000e+00
166929	Britax Pavilion 70-G3 Convertible Car Seat Sea...	We LOVE this seat! As parents to 8 children ra...	5	We LOVE this seat As parents to 8 children ran...	1	1.000000e+00
76549	Britax Advocate 65 CS Click & Safe Convert...	The Britax Advocate CS appears similar to the ...	4	The Britax Advocate CS appears similar to the ...	1	1.000000e+00
50735	Joovy Zoom 360 Swivel Wheel Jogging Stroller, ...	The joovy zoom 360 was the perfect solution fo...	5	The joovy zoom 360 was the perfect solution fo...	1	1.000000e+00
103186	Thirsties Duo Wrap Snap, Ocean Blue, Size One ...	I am reviewing these covers because reading re...	5	I am reviewing these covers because reading re...	1	1.000000e+00
129722	Bumbleride 2011 Flite Lightweight Compact Trav...	This is a review of the 2012 Bumbleride Flite ...	5	This is a review of the 2012 Bumbleride Flite ...	1	1.000000e+00

50026 rows × 6 columns

Compute accuracy of the classifier

test_matrix = vectorizer.transform(test_data['review_clean'].values.astype('U'))
scores_test = sentiment_model.predict(test_matrix)
print(scores_test)

[ 1  1  1 ... -1  1  1]

type(scores_test)

numpy.ndarray

test_data['sentiment']

103733    1
157975    1
127693    1
113718   -1
20416     1
62236    -1
93741     1
182819    1
108942    1
63171     1
27972     1
123316    1
60874     1
89381     1
146248    1
167897    1
129689    1
54150     1
8352      1
55700     1
83479     1
93618     1
130762   -1
73991     1
174021    1
17830     1
62700     1
33106     1
156734   -1
67422     1
         ..
29152     1
145868    1
133924    1
144259    1
155251    1
49516     1
139738    1
155993    1
28007     1
124413   -1
114981    1
115743   -1
109348    1
73490    -1
10461     1
120390   -1
56823     1
110189    1
151692    1
134663    1
14881     1
106787    1
9918      1
79920     1
31318     1
54722    -1
106191    1
79222    -1
93173    -1
86465     1
Name: sentiment, Length: 50026, dtype: int64

len(scores_test)

50026

len(test_data['sentiment'])

50026

type(test_data['sentiment'])

pandas.core.series.Series

test_data['sentiment'].index

Int64Index([103733, 157975, 127693, 113718,  20416,  62236,  93741, 182819,
            108942,  63171,
            ...
             14881, 106787,   9918,  79920,  31318,  54722, 106191,  79222,
             93173,  86465],
           dtype='int64', length=50026)

test_data['sentiment'].values

array([ 1,  1,  1, ..., -1, -1,  1], dtype=int64)

type(test_data['sentiment'].values)

numpy.ndarray

accuracy = (scores_test == test_data['sentiment'].values)

accuracy

array([ True,  True,  True, ...,  True, False,  True])

len(accuracy)

50026

correct_counts = np.count_nonzero(accuracy == True)

accuracy_math = correct_counts/len(accuracy)

accuracy_math

0.9300163914764322

Learn another classifier with fewer words

significant_words = ['love', 'great', 'easy', 'old', 'little', 'perfect', 'loves', 
      'well', 'able', 'car', 'broke', 'less', 'even', 'waste', 'disappointed', 
      'work', 'product', 'money', 'would', 'return']

vectorizer_word_subset = CountVectorizer(vocabulary=significant_words) # limit to 20 words
train_matrix_word_subset = vectorizer_word_subset.fit_transform(train_data['review_clean'].values.astype('U'))
test_matrix_word_subset = vectorizer_word_subset.transform(test_data['review_clean'].values.astype('U'))

Train a logistic regression model on a subset of data

simple_model = linear_model.LogisticRegression()

simple_model.fit(train_matrix_word_subset, train_data['sentiment'])

LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,
          verbose=0, warm_start=False)

simple_model.coef_

array([[ 1.34240752,  0.93331879,  1.17057657,  0.09645538,  0.49546276,
         1.51007107,  1.70968884,  0.48996502,  0.18266183,  0.06213742,
        -1.63855688, -0.16152094, -0.52904125, -2.01345347, -2.36957333,
        -0.64400431, -0.32167306, -0.90190808, -0.33086295, -2.06984646]])

simple_model.intercept_

array([1.30862795])

exclude the intercept (w0)

['love', 'great', 'easy', 'old', 'little', 'perfect', 'loves',

'well', 'able', 'car', 'broke', 'less', 'even', 'waste', 'disappointed',

'work', 'product', 'money', 'would', 'return']

simple_model_noint = linear_model.LogisticRegression(fit_intercept=False)

simple_model_noint.fit(train_matrix_word_subset, train_data['sentiment'])

LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=False,
          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,
          verbose=0, warm_start=False)

simple_model_noint.coef_

array([[ 1.97893948,  1.5148022 ,  1.69499344,  0.49472526,  0.91975043,
         2.17222529,  2.41423013,  0.94998025,  0.4492522 ,  0.22932683,
        -1.41097026,  0.10142036, -0.27978831, -1.86577283, -2.13991375,
        -0.41593736,  0.01620127, -0.61563086, -0.06504618, -1.93940329]])

simple_model_noint.intercept_

0.0

Comparing models

First, compute the classification accuracy on the train_data

def model_accuracy(model, input_matrix, target_parameter):
    recall = model.predict(input_matrix)
    TF_array = (recall == target_parameter)
    correct_counts = np.count_nonzero(TF_array == True)
    accu_math = correct_counts/len(TF_array)
    return accu_math

model_accuracy(sentiment_model, train_matrix, train_data['sentiment'])

0.9689529325086099

model_accuracy(simple_model, train_matrix_word_subset, train_data['sentiment'])

0.867373164504909

model_accuracy(simple_model_noint, train_matrix_word_subset, train_data['sentiment'])

0.7785411990473416

## First, compute the classification accuracy on the test_data

model_accuracy(sentiment_model, test_matrix, test_data['sentiment'])

0.9300163914764322

model_accuracy(simple_model, test_matrix_word_subset, test_data['sentiment'])

0.867269020109543

model_accuracy(simple_model_noint, test_matrix_word_subset, test_data['sentiment'])

0.7796545796185983

test numpy sort function here

list1 = [[4,3,2],[2,1,4],[1,1,2],[4,1,3]]
array=np.array(list1)

array

array([[4, 3, 2],
       [2, 1, 4],
       [1, 1, 2],
       [4, 1, 3]])

array.sort(axis=1)

array

array([[2, 3, 4],
       [1, 2, 4],
       [1, 1, 2],
       [1, 3, 4]])

array.sort(axis=0)

array

array([[1, 1, 2],
       [1, 2, 4],
       [1, 3, 4],
       [2, 3, 4]])

len(array)

4

array[-3:]

array([[1, 2, 4],
       [1, 3, 4],
       [2, 3, 4]])

Test ndarrary to pandas series

test_data = proba

test_data

array([[3.42077940e-03, 9.96579221e-01],
       [1.22103669e-07, 9.99999878e-01],
       [2.21610865e-02, 9.77838914e-01]])

type(test_data)

numpy.ndarray

2nd ([0] -> [1]) element of all the rows

test_positive_possibility = test_data[:,1]

test_positive_possibility

array([0.99657922, 0.99999988, 0.97783891])

type(sample_test_data)

pandas.core.frame.DataFrame

sample_test_data

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	name	review	rating	review_clean	sentiment
27972	Bumkins Waterproof Superbib, Blue Fizz	I love these bibs! We have about 8 of them in...	5	I love these bibs We have about 8 of them in ...	1
123316	Safety 1st High Door Lock	My 4 year old gets up earlier than me, this me...	5	My 4 year old gets up earlier than me this mea...	1
60874	DadGear Courier Diaper Bag - Orange Retro Stripe	love the bag, especially since it's over the s...	4	love the bag especially since its over the sho...	1

sample_test_data.loc[:,'probability'] = test_positive_possibility

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\indexing.py:357: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self.obj[key] = _infer_fill_value(value)
C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\indexing.py:537: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self.obj[item] = s

# sample_test_data['probability'] = test_positive_possibility

sample_test_data

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	name	review	rating	review_clean	sentiment	probability
27972	Bumkins Waterproof Superbib, Blue Fizz	I love these bibs! We have about 8 of them in...	5	I love these bibs We have about 8 of them in ...	1	0.996579
123316	Safety 1st High Door Lock	My 4 year old gets up earlier than me, this me...	5	My 4 year old gets up earlier than me this mea...	1	1.000000
60874	DadGear Courier Diaper Bag - Orange Retro Stripe	love the bag, especially since it's over the s...	4	love the bag especially since its over the sho...	1	0.977839

define series index and columns

df = DataFrame(test_data,
              index = ['one', 'two', 'three'],
              columns = ['negative', 'positive'])

df

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	negative	positive
one	3.420779e-03	0.996579
two	1.221037e-07	1.000000
three	2.216109e-02	0.977839

sample_test_data

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	name	review	rating	review_clean	sentiment	probability
27972	Bumkins Waterproof Superbib, Blue Fizz	I love these bibs! We have about 8 of them in...	5	I love these bibs We have about 8 of them in ...	1	0.996579
123316	Safety 1st High Door Lock	My 4 year old gets up earlier than me, this me...	5	My 4 year old gets up earlier than me this mea...	1	1.000000
60874	DadGear Courier Diaper Bag - Orange Retro Stripe	love the bag, especially since it's over the s...	4	love the bag especially since its over the sho...	1	0.977839

df_1 = DataFrame(test_data,
              columns = ['negative', 'positive'])

df_1

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	negative	positive
0	3.420779e-03	0.996579
1	1.221037e-07	1.000000
2	2.216109e-02	0.977839