David’s PIC16B Blog

Final Project

2022-06-08T00:00:00+00:00

PIC 16B Project Group Submission

1. Introduction

We wanted to find out whether or not quarterly earnings conferences give any extra insight into how a company is doing financially compared to the raw data. To this end, we regressed daily price changes with different aspects of earnings reports with linear regression and neural network models, and did a sentiment analysis on earnings conferences to see if either one has any relation, or if one has a stronger relation with future changes in value than the other. Here is a link to our GitHub page for the project.

Flowchart:

2.1 Data Imports and Organization

To build our working dataframe we used the SimFin API to pull the (numeric) financial statements for all available quarterly reports.

# Set your SimFin+ API-key for downloading data.
sf.set_api_key('H9OLpPRQ36sXqfhoOtyBb0Kwv4Dw0q3W')
 
# Set the local directory where data-files are stored.
# The directory will be created if it does not already exist.
sf.set_data_dir('~/simfin_data/')
 
# Download the data from the SimFin server and load into a Pandas DataFrame.
quarterly = sf.load_income(variant='quarterly', market='us')
 
# Print the first rows of the data.
#print(quarterly.head())
quarterly.head()

We then repeated the same procedure to build a company df containing SimFinId, company name, and sector info:

 
# Download the data from the SimFin server and load into a Pandas DataFrame.
companies = sf.load_companies(market='us')

and then the same for a translational dataframe between industry id and industry name

 
# Download the data from the SimFin server and load into a Pandas DataFrame.
industries = sf.load_industries()
print(industries["Sector"])

From here, we needed to build a df of stock price data (open/close price, volume, etc.):

 
# Download the data from the SimFin server and load into a Pandas DataFrame.
shares = sf.load_shareprices(variant='daily', market='us')
 
# Print the first rows of the data.
print(shares.head())
#print(shares["Date"].head())
print(shares.columns)
print(shares.index)
#so date is a part of a multi index, not a column
#lets reset that index to get these as columns for easier bool syntax
shares = shares.reset_index()
#see if that worked
shares.columns
#lets examine the types for any future comparison issues
print(shares.dtypes)
#okay so its a datetime, need to make sure other dates are in this format
#pd.to_numeric(shares.Date) - incase the other isn't, convert to float
print(shares.head())

We then merged these on the appropriate report release dates for each ticker for each report to create a single, unified, working pandas dataframe, from which we calculated our target variable “DeltaPrice” (the difference between close and next day opening price as a measure of after hours price movement in response to a given quarterly report) and populated a new column with it:

# Combining databases
 
merge1 = pd.merge(quarterly, companies_tick, how = 'left', on = 'SimFinId')
 
alldata = pd.merge(merge1, industries, how = 'left', on = 'IndustryId')
 
print(alldata.head())
 
#lets see what our types are
Alldata.dtypes
y(lambda row: row["NextDay"] - row["Close"], axis=1)

From here, all that remains data handling wise, is specific to each application going forward (e.g. dropping unneeded columns or encoding for LR)```

shares["NextDay"] = shares["Open"].shift(-1)
#alldata.rename(columns ={'Publish Date': 'Date'}) - figure out why rename isn't recognizing Publish Date
 
#hacky short term solution
alldata["Date"] = alldata["Publish Date"]
 
shares2 = shares[["SimFinId", "Open", "Close", "Date", "NextDay"]]
alldata = pd.merge(alldata, shares2, how = "left", on = ["SimFinId", "Date"])
#cool, now we have everything together, I'll clean this up to only be what we need instead of a bulk merge later
 
 
alldata["DeltaPrice"] = alldata.appl

2.2 Regression

To start with the numerical analysis, we created a linear regression model, which took in the numerical values from the dataset, and regressed them onto our variable ‘DeltaPrice’, which we created by subtracting the close date from the open date of the next day for each datapoint. We then used sklearn’s LinearRegression model to regress the data, and put all of this, as well as a train-test-split testing mechanism into a function that takes in a subset of our main dataframe to perform the linear regression. With the whole dataset, the linear regression performed at around 31% accuracy, which we considered to be good, since it is predicting the exact change in price, for which there are many values. Below is the code for the function:

def lr(df = newdata):
  mlinx = []
  mliny = []
  regx = df.drop("DeltaPrice", axis = 1)
  for i in df["DeltaPrice"]:
    mliny.append(round(i, 2))
    mlinx.append([])
  for i in regx.columns:
    k = 0
    for j in regx[i]:
      mlinx[k].append(j)
      k += 1
  
  X2 = sm.add_constant(mlinx)
  est = sm.OLS(mliny, X2)
  est2 = est.fit()
  print(est2.summary())
  j = 1
  for i in regx.columns:
    print(str(j) + ": " + i)
    j += 1
  X_train, X_test, y_train, y_test = train_test_split(mlinx, mliny, test_size = 0.2, random_state = 42)
  LR = LinearRegression().fit(X_train, y_train)
  print("Test score:")
  print(LR.score(X_train, y_train))
  return X_train, y_train

For the neural networks, we used a similar format as for the linear regression model, but we changed it so that the user can input which sector they would like to run the regression on, since the model is too big to run on the whole data, and we got better results from running it on just one sector than on the whole dataset, and whether they would like to use the model with or without the LSTM, or long short term memory, layer. We found that the model works moderately well without the LSTM, but produces meaningless results with the LSTM layer, so the default setting is without the LSTM layer. Below is the code for the function, and a screenshot of some of the last epochs of the model run without LSTM on the sector ‘Healthcare’, which have a loss of around 20.

def nn(sector="Technology", rnn = False):
  rnndf = regdata[regdata["Sector"] == sector]
  rnndf = rnndf.drop(["Sector", "Industry", "Date", "NextDay"], axis = 1)
  rnndf = rnndf[rnndf["Close"] != 0]
  rnndf["PctChg"] = rnndf.apply(lambda row: round(1000 * (row["DeltaPrice"] / row["Close"])) / 10, axis=1)
  rnndf = rnndf[rnndf["PctChg"] < 70]
  rnndf = rnndf[rnndf["PctChg"] > -70]
  rnnx = rnndf.drop(["DeltaPrice", "PctChg"], axis = 1)
  cmp = rnndf["SimFinId"]
  cmp = cmp.drop_duplicates()
  ryl = []
  rny = rnndf.sort_values("NewDate", axis = 0)
  for i in rny["PctChg"]:
    ryl.append(round(i, 2))
  rxl = []
  rnx = rnnx.sort_values("NewDate")
  for i in rnx["NewDate"]:
    rxl.append([])
  for i in rnx.columns:
    k = 0
    for j in rnx[i]:
      rxl[k].append(j)
      k += 1
  rxa = np.array(rxl)
  rya = np.array(ryl)
  if rnn:
    model.fit(rxa, rya, epochs=50, verbose=1)
  else:
    model1.fit(rxa, rya, epochs=500, verbose=1)

2.3 Scraper

To gather transcripts, we used the Financial Modeling Prep API. We first built a function to structure and individual query:

def call_pull(ticker, year, quarter,key = '8dccdfa88c60658c4170dbaa660a6c5c',
             date=False):
   transcript = requests.get(f'https://financialmodelingprep.com/api/v3/earning_call_transcript/{ticker}?quarter={quarter}&year={year}&apikey={key}').json()
   tran = transcript[0]['content']
   date = transcript[0]['date']
   if (date):
     return [tran, date]
   else:
     return tran

Next we build a function to automate the pulls for each entry in a provided data frame:

def transcript_fetch(df):
 
 df["Fiscal Period"].str.slice(1,1)
 
 #make transcript column
 df['Transcript'] = ""
 
 #start a counter
 count = 0
 
 # for each company ticker
 for ticker in df["Ticker"]:
   #for each call date
   for date in df["Publish Date"][df["Ticker"] == ticker]:
     #get quarter from fiscal period
     for period in (tech_sample["Fiscal Period"][(tech_sample["Ticker"] == ticker) & (tech_sample["Publish Date"] == date)]):   
     #pull numeric value only
       quarter = period[1]
  
     #pull year
     #month = date.month
     year = date.year
 
     count +=1
    
     #pull transcript
     try:
       trans = call_pull(ticker, str(year), str(quarter))
    
       #convert to string
       trans = str(trans[0])
 
     #if pull fails
     except:
       trans = np.NAN #make NAN for later drop
 
     #add to "Transcript" column
     tech.loc[((tech["Publish Date"] == date) & (tech["Ticker"] == ticker), 'Transcript')] = trans
 
 #return df now with transcript strings
 return df

Notably, the Fiscal Period returned by the API is actually a not-too-well-documented object, not a string, and so required an uncomfortable amount of hacky syntax to coax say, the numeric only “1” string out of a Q1 fiscal period OBJ.

2.4 Sentiment Analysis

From here, largely on account of our late-dropping group member, we had to curtail our pulls to random samples (n=1000 and n=2000). However, we did have nice syntax prepared to tokenize and core the text using our sentiment dictionaries and add these elements as new rows to our dataframe:

import pandas as pd
from nltk.tokenize import WordPunctTokenizer as wpt

import nltk
 
nltk.download("opinion_lexicon")
 
from nltk.corpus import opinion_lexicon
 
pos_list=list(opinion_lexicon.positive())
neg_list=list(opinion_lexicon.negative())
print(pos_list)

#for each transcript
for transcript in final_sample["Transcript"]:
 
 # Tokenize the words and convert to lowercase
 text = transcript.split()
 text = [text.lower() for text in text]
 
 # Remove the stopwords
 text_clean = [word for word in text if word not in stopwords]
 
 words_pos = [word for word in text_clean if word in pos_list]
 words_neg = [word for word in text_clean if word in neg_list]
 
 pos_score = len(words_pos)
 neg_score = len(words_neg)
 
 tokens = lm.tokenize(transcript)
 score = lm.get_score(tokens)
 
 final_sample.loc[((final_sample["Transcript"] == transcript), 'Positive')] = pos_score
 final_sample.loc[((final_sample["Transcript"] == transcript), 'Negative')] = neg_score
 final_sample.loc[((final_sample["Transcript"] == transcript), 'Ratio')] = pos_score / neg_score
 

We then used seaborn to plot the relationship between positive, negative, and pos/neg ratio scores and our target variable of DeltaPrice:

We also plotted the technology sector in isolation and noticed a curious group of points where “no bad news” seemed to be more strongly correlated with DeltaPrice than actual positive sentiment or even the ratio of positive to negative sentiment:

3 Conclusion

We found that recurrent neural networks do not work very well on stock data, and that our linear regression model does surprisingly well. On the sentiment side, the plotted results were very much undermined by the lack of time to run full 36,000+ row pulls. The random samples undermined any patterns found within the time series of a particular company for both the sentiment analysis and the neural networks. Ideally, moving forward we would run a complete S&P 500 pull (to avoid potential noise from small-caps), and precluding reports from the period of extreme Fed Quantitative Easing in response to COVID (or isolating and analyzing them separately). Similarly, we would like to follow through with our original intent to both run a neural network model on vectorized transcript text and to see the effect of the inclusion of sentiment scores as features within our most successful numeric model.

HW4 - Fake News Classifier

2022-05-16T00:00:00+00:00

Classifying Misinformation with NLP

For this assignment we’re going to develop a misinformation (colloqually “fake news”) classifier using TensorFlow.

Our data for this assignment comes from the following article:

Ahmed H, Traore I, Saad S. (2017) “Detection of Online Fake News Using N-Gram Analysis and Machine Learning Techniques. In: Traore I., Woungang I., Awad A. (eds) Intelligent, Secure, and Dependable Systems in Distributed and Cloud Environments. ISDDC 2017. Lecture Notes in Computer Science, vol 10618. Springer, Cham (pp. 127-138).

And is found by way of Kaggle. Here we’re going to use a pre-split version of this data as graceously provided by UCLA Mathmatics Professor, Phil Chodrow, using the link found below.

train_url = "https://github.com/PhilChodrow/PIC16b/blob/master/datasets/fake_news_train.csv?raw=true"

Now let’s get the slew of imports handled:

#import the usual suspects
import pandas as pd
import numpy as np
import tensorflow as tf

#import string + regular expressions...just in case, don't presume base pkgs...
import re
import string

#import keras, layers, loss functions etc 
#ignore the yellow warning, currently a bug even on colab...a google product
from tensorflow.keras import layers
from tensorflow.keras import losses
from tensorflow import keras

#imports for text handling (vectorization etc.)
from tensorflow.keras.layers.experimental.preprocessing import TextVectorization
from tensorflow.keras.layers.experimental.preprocessing import StringLookup

#for visualizing model layers
from tensorflow.keras import utils

#imports for encoding and our splits, the sklearn classics
from sklearn.preprocessing import LabelEncoder
#used if you prefer to split at the df rather than ds stage
from sklearn.model_selection import train_test_split

# for simple visualizations (line graphs)
from matplotlib import pyplot as plt

# for embedding visualizations
from sklearn.decomposition import PCA
import plotly.express as px 
import plotly.io as pio
pio.templates.default = "plotly_white"

From here let’s read in the data:

#csv read in via pandas
raw_df = pd.read_csv(train_url)

Let’s take a look at what we have:

raw_df

	Unnamed: 0	title	text	fake
0	17366	Merkel: Strong result for Austria's FPO 'big c...	German Chancellor Angela Merkel said on Monday...	0
1	5634	Trump says Pence will lead voter fraud panel	WEST PALM BEACH, Fla.President Donald Trump sa...	0
2	17487	JUST IN: SUSPECTED LEAKER and “Close Confidant...	On December 5, 2017, Circa s Sara Carter warne...	1
3	12217	Thyssenkrupp has offered help to Argentina ove...	Germany s Thyssenkrupp, has offered assistance...	0
4	5535	Trump say appeals court decision on travel ban...	President Donald Trump on Thursday called the ...	0
...	...	...	...	...
22444	10709	ALARMING: NSA Refuses to Release Clinton-Lynch...	If Clinton and Lynch just talked about grandki...	1
22445	8731	Can Pence's vow not to sling mud survive a Tru...	() - In 1990, during a close and bitter congre...	0
22446	4733	Watch Trump Campaign Try To Spin Their Way Ou...	A new ad by the Hillary Clinton SuperPac Prior...	1
22447	3993	Trump celebrates first 100 days as president, ...	HARRISBURG, Pa.U.S. President Donald Trump hit...	0
22448	12896	TRUMP SUPPORTERS REACT TO DEBATE: “Clinton New...	MELBOURNE, FL is a town with a population of 7...	1

22449 rows × 4 columns

Each row of the data corresponds to an article. The title column gives the title of the article, while the text column gives the full article text. The final column, called fake, is 0 if the article is true and 1 if the article contains fake news, as determined by the authors of the paper above.

Making the Dataset

Next we’re going to write a function called make_dataset that will remove stop words—uninformative words such as “the,” “and,” or “but”—and return a tf.data.Dataset with two inputs and one output. The input will be the title and text, the output will be the fake column (0/1).

Thankfully, we can just use the stop words from Natural Language Toolkit (NLTK). Often you may want to use a specialized set of stop words for a specific application, but we’re working with very standard, formal English, and NLTK is about as default as it gets, so let’s not make things hard on ourselves.

The stop word dl/import:

import nltk
nltk.download('stopwords')
from nltk.corpus import stopwords

[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data]   Unzipping corpora/stopwords.zip.

The function itself:

def make_dataset(in_df):
  """
  Function to remove stop words from the provided pandas dataframe of article
  titles and text, then create a  TensorFlow Dataset from this dataframe.

  Parameters
  ----------
  in_df: the supplied pandas dataframe to process.

  Return
  ------
  out_ds: the output TensorFlow Dataset.
  """

  #Pythonic Quibbles - everything is by ref and const isn't a thing so let's not 
  #presume the argument df is to be modified...our copy will be deleted at the
  #end of function scope anyway. An original df and a new ds seems more logical
  #than a new ds and a df unknowingly modified by a blackbox.
  new_df = in_df.copy()

  #assign our imported stopwords to a handy variable
  stop = stopwords.words('english')

  #remove stopwords from article titles and texts  
  new_df[['title','text']].apply(lambda x: [item for item in x if 
                                           item not in stop])
  
  #Construct the tf.data.Dataset
  out_ds = tf.data.Dataset.from_tensor_slices(
    ( { "title": new_df[["title"]], "text": new_df['text'] },#dict to map inputs 
      { "fake": new_df[["fake"]] } )) #dict to map output

  #shuffle dataset
  out_ds = out_ds.shuffle(buffer_size = len(out_ds))
  
  #batch dataset
  out_ds = out_ds.batch(100)

  #return finished dataset
  return out_ds

#actually make the dataset using our function
data = make_dataset(raw_df)

Validation Data

Next we’ll split off 20% of our training set for validation

train_size = int(0.8*len(data)) 
val_size = int(0.2*len(data))

train = data.take(train_size) # data[:train_size]
# data[train_size : train_size + val_size]
val = data.skip(train_size).take(val_size) 

len(train), len(val)

(180, 45)

Cool, so our 22,449 rows (articles) are split into 180 training batches and 45 validation batches.

Base Rate

There are plently of easy ways find the base rate of fake articles in our original dataframe.

For instance, take the mean of the “fake” column (0 or 1):

raw_df["fake"].mean()

0.522963160942581

Or group by and compare the sizes of the two groups:

raw_df.groupby("fake").size()

fake
0    10709
1    11740
dtype: int64

11740 / (11740 + 10709)

0.522963160942581

If we want to do this for our post train/val split dataset it’s a tad more complicated. Our split was performed on a shuffled dataset of over 22,000 articles, so realistically, any random split should not vary much from the fake rate seen in the original, complete dataset. We’re working with large enough numbers that the noise shouldn’t be too drastic. E.g. in an 80/20 random split with n=20, a single article falling on oneside of the line or the other is the difference between a validation set thats 3/4th’s fake vs 2/4th’s fake…we don’t have that problem here with n=22449.

However, to be properly thorough, let’s take a look:

#unbatch our train set
unbatched = train.unbatch()
#map our label back to a bool (0/1)
unbatched = unbatched.map(lambda x, label: label['fake'][0])
#cast to numpy itr then to list
np_data = list(unbatched.as_numpy_iterator())
#normal math as per before
( sum(np_data) ) / (len(np_data))

0.5227222222222222

So as expected, our base rate for the training set is very close to that of the total dataset. If you rerun the split a few cells back and test the difference in the training set rate, you’ll see that it will line up with the original base rate down to the 3rd or 4th decimal place the vast majority of the time.

If the proprietary-ness of the TensorFlow syntax feels a tad obtuse to you, then take it as another point for doing as much as you can in pandas first. Here, working with a complete data set from the get-go that is of a size to reasonably read into memory all at once, you can certainly get away with performing your splits using sklearn at the pandas dataframe stage prior to the dataset conversion. That said, it’s always good to get practice navigating TensorFlow syntax and the somewhat oddly formatted documentation, since there will eventually be things you’ll heavily lean on prefetching for, where the data isn’t an easy single csv to df read-in.

Preprocessing: Standardization

Next we’ll standardize the text by converting it all to lowercase and removing punctuation.

Note: Punctuation is often removed when working large corpora and you want to do some specific task like document similarity or classification, but you would likely want to preserve it for an SA model of tweets or reviews…if an exclamation mark is one of only 280 characters, it’s likely pretty important! However, we’re classifying 22,449 articles, so once gain, we’re playing it very by the book and removing punctuation.

def standardization(input_data):
  """
  Function to standardize text via the conversion to all-lower-case and the 
  removal of punctuation

  Parameters
  ----------
  input_data: a TensorFlow dataset, the text to be standardized

  Return
  ------
  no_punctuation - the dataset w/o punctuation and capitalization
  """

  #convert to lowercase 
  lowercase = tf.strings.lower(input_data)
  #remove punctuation
  no_punctuation = tf.strings.regex_replace(lowercase,
                                '[%s]' % re.escape(string.punctuation),'')
  return no_punctuation 

Vectorization

Next, we’ll need to represent the text as a vector, replacing words with their frequency rank within the dataset.

First we’ll create a text vectorization layer for our model:

#preparing a text vectorization layer for tf model
size_vocabulary = 2000

#title vectorization layer
vectorize_layer = TextVectorization(
    standardize=standardization, #invoke our standardization function
    max_tokens=size_vocabulary, # only consider this many words
    output_mode='int', #get frequency ranking for each word in the training ds
    output_sequence_length=500) #turn each

Now we’ll “adapt” our layer to learn to map words based on the two components of our training data (title and text):

vectorize_layer.adapt(train.map(lambda x, y: x["title"]))
vectorize_layer.adapt(train.map(lambda x, y: x["text"]))

Creating Models

First we need to create two keras.Input’s, one for our titles and one for our text. Both have a shape of (1, )—there’s only a single title or text in each. We also need to specify their type (both are string) and name (same as their dictionary key in the dataset).

title_input = keras.Input(
    shape=(1,),
    name = "title", # same name as the dictionary key in the dataset
    dtype = "string"
)

text_input = keras.Input(
    shape=(1,),
    name = "text", # same name as the dictionary key in the dataset
    dtype = "string"
)

First Model

Now we’re ready to create our first model, which will only use the title to classify the article as fake or not. We’ll use the functional API to assembly our layers:

title_features = vectorize_layer(title_input) #vectorize text input
title_features = layers.Embedding(size_vocabulary, output_dim = 3, name="embedding")(title_features)
title_features = layers.Dropout(0.2)(title_features) #dropout for overfitting
title_features = layers.GlobalAveragePooling1D()(title_features)
title_features = layers.Dropout(0.2)(title_features) #dropout for overfitting
title_features = layers.Dense(32, activation='relu')(title_features)

# output layer
title_output = layers.Dense(2, name = "fake")(title_features)

Now let’s put it all together and take a look:

model1 = keras.Model(
      inputs = title_input,
      outputs = title_output
)

model1.summary()

Model: "model"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 title (InputLayer)          [(None, 1)]               0         
                                                                 
 text_vectorization (TextVec  (None, 500)              0         
 torization)                                                     
                                                                 
 embedding (Embedding)       (None, 500, 3)            6000      
                                                                 
 dropout (Dropout)           (None, 500, 3)            0         
                                                                 
 global_average_pooling1d (G  (None, 3)                0         
 lobalAveragePooling1D)                                          
                                                                 
 dropout_1 (Dropout)         (None, 3)                 0         
                                                                 
 dense (Dense)               (None, 32)                128       
                                                                 
 fake (Dense)                (None, 2)                 66        
                                                                 
=================================================================
Total params: 6,194
Trainable params: 6,194
Non-trainable params: 0
_________________________________________________________________

Or if you want to visualize it:

keras.utils.plot_model(model1)

Now let’s compile it:

model1.compile(optimizer="adam",
              loss = losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=["accuracy"])

Train it:

history1 = model1.fit(train, 
                    validation_data=val,
                    epochs = 20)

Epoch 1/20


/usr/local/lib/python3.7/dist-packages/keras/engine/functional.py:559: UserWarning: Input dict contained keys ['text'] which did not match any model input. They will be ignored by the model.
  inputs = self._flatten_to_reference_inputs(inputs)


180/180 [==============================] - 5s 7ms/step - loss: 0.6924 - accuracy: 0.5181 - val_loss: 0.6911 - val_accuracy: 0.5354
Epoch 2/20
180/180 [==============================] - 1s 7ms/step - loss: 0.6902 - accuracy: 0.5247 - val_loss: 0.6867 - val_accuracy: 0.5314
Epoch 3/20
180/180 [==============================] - 1s 7ms/step - loss: 0.6729 - accuracy: 0.6078 - val_loss: 0.6432 - val_accuracy: 0.8633
Epoch 4/20
180/180 [==============================] - 1s 7ms/step - loss: 0.5800 - accuracy: 0.7944 - val_loss: 0.4979 - val_accuracy: 0.8523
Epoch 5/20
180/180 [==============================] - 1s 7ms/step - loss: 0.4315 - accuracy: 0.8739 - val_loss: 0.3557 - val_accuracy: 0.9004
Epoch 6/20
180/180 [==============================] - 1s 7ms/step - loss: 0.3383 - accuracy: 0.8834 - val_loss: 0.2889 - val_accuracy: 0.8998
Epoch 7/20
180/180 [==============================] - 1s 7ms/step - loss: 0.2960 - accuracy: 0.8879 - val_loss: 0.2411 - val_accuracy: 0.9155
Epoch 8/20
180/180 [==============================] - 1s 7ms/step - loss: 0.2703 - accuracy: 0.8991 - val_loss: 0.2238 - val_accuracy: 0.9204
Epoch 9/20
180/180 [==============================] - 1s 7ms/step - loss: 0.2506 - accuracy: 0.9027 - val_loss: 0.2052 - val_accuracy: 0.9283
Epoch 10/20
180/180 [==============================] - 1s 7ms/step - loss: 0.2393 - accuracy: 0.9071 - val_loss: 0.2045 - val_accuracy: 0.9204
Epoch 11/20
180/180 [==============================] - 1s 7ms/step - loss: 0.2327 - accuracy: 0.9094 - val_loss: 0.1969 - val_accuracy: 0.9249
Epoch 12/20
180/180 [==============================] - 1s 7ms/step - loss: 0.2245 - accuracy: 0.9117 - val_loss: 0.1900 - val_accuracy: 0.9285
Epoch 13/20
180/180 [==============================] - 1s 6ms/step - loss: 0.2156 - accuracy: 0.9158 - val_loss: 0.1765 - val_accuracy: 0.9314
Epoch 14/20
180/180 [==============================] - 1s 6ms/step - loss: 0.2068 - accuracy: 0.9194 - val_loss: 0.1763 - val_accuracy: 0.9305
Epoch 15/20
180/180 [==============================] - 1s 7ms/step - loss: 0.2068 - accuracy: 0.9204 - val_loss: 0.1746 - val_accuracy: 0.9317
Epoch 16/20
180/180 [==============================] - 1s 6ms/step - loss: 0.2017 - accuracy: 0.9216 - val_loss: 0.1695 - val_accuracy: 0.9373
Epoch 17/20
180/180 [==============================] - 1s 7ms/step - loss: 0.1976 - accuracy: 0.9241 - val_loss: 0.1615 - val_accuracy: 0.9373
Epoch 18/20
180/180 [==============================] - 1s 6ms/step - loss: 0.1939 - accuracy: 0.9244 - val_loss: 0.1612 - val_accuracy: 0.9377
Epoch 19/20
180/180 [==============================] - 1s 6ms/step - loss: 0.1898 - accuracy: 0.9256 - val_loss: 0.1594 - val_accuracy: 0.9386
Epoch 20/20
180/180 [==============================] - 1s 6ms/step - loss: 0.1911 - accuracy: 0.9265 - val_loss: 0.1492 - val_accuracy: 0.9481

So, we’re hitting mid 90’s for accuracy on our validation set just on title alone! Validation performance actually slightly exceed training performance…guess we’re not overfitting!

Second Model

For this model we’re going to follow the same proceedure, except using only the article text rather than only the article title.

text_features = vectorize_layer(text_input) #vectorize text input
text_features = layers.Embedding(size_vocabulary, output_dim = 3, name="embedding2")(text_features)
text_features = layers.Dropout(0.2)(text_features) #dropout for overfitting
text_features = layers.GlobalAveragePooling1D()(text_features)
text_features = layers.Dropout(0.2)(text_features) #dropout for overfitting
text_features = layers.Dense(32, activation='relu')(text_features)

# output layer
text_output = layers.Dense(2, name = "fake")(text_features)

model2 = keras.Model(
      inputs = text_input,
      outputs = text_output
)

model2.summary()

Model: "model_4"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 text (InputLayer)           [(None, 1)]               0         
                                                                 
 text_vectorization (TextVec  (None, 500)              0         
 torization)                                                     
                                                                 
 embedding2 (Embedding)      (None, 500, 3)            6000      
                                                                 
 dropout_4 (Dropout)         (None, 500, 3)            0         
                                                                 
 global_average_pooling1d_2   (None, 3)                0         
 (GlobalAveragePooling1D)                                        
                                                                 
 dropout_5 (Dropout)         (None, 3)                 0         
                                                                 
 dense_3 (Dense)             (None, 32)                128       
                                                                 
 fake (Dense)                (None, 2)                 66        
                                                                 
=================================================================
Total params: 6,194
Trainable params: 6,194
Non-trainable params: 0
_________________________________________________________________

keras.utils.plot_model(model2)

model2.compile(optimizer="adam",
              loss = losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=["accuracy"])

history2 = model2.fit(train, 
                    validation_data=val,
                    epochs = 20)

Epoch 1/20


/usr/local/lib/python3.7/dist-packages/keras/engine/functional.py:559: UserWarning: Input dict contained keys ['title'] which did not match any model input. They will be ignored by the model.
  inputs = self._flatten_to_reference_inputs(inputs)


180/180 [==============================] - 3s 15ms/step - loss: 0.6566 - accuracy: 0.6387 - val_loss: 0.5718 - val_accuracy: 0.8620
Epoch 2/20
180/180 [==============================] - 3s 14ms/step - loss: 0.4307 - accuracy: 0.8971 - val_loss: 0.2919 - val_accuracy: 0.9445
Epoch 3/20
180/180 [==============================] - 3s 14ms/step - loss: 0.2416 - accuracy: 0.9411 - val_loss: 0.1863 - val_accuracy: 0.9602
Epoch 4/20
180/180 [==============================] - 3s 14ms/step - loss: 0.1755 - accuracy: 0.9562 - val_loss: 0.1436 - val_accuracy: 0.9665
Epoch 5/20
180/180 [==============================] - 3s 14ms/step - loss: 0.1460 - accuracy: 0.9612 - val_loss: 0.1191 - val_accuracy: 0.9751
Epoch 6/20
180/180 [==============================] - 3s 14ms/step - loss: 0.1244 - accuracy: 0.9694 - val_loss: 0.1038 - val_accuracy: 0.9751
Epoch 7/20
180/180 [==============================] - 3s 14ms/step - loss: 0.1117 - accuracy: 0.9718 - val_loss: 0.0898 - val_accuracy: 0.9757
Epoch 8/20
180/180 [==============================] - 3s 14ms/step - loss: 0.1017 - accuracy: 0.9723 - val_loss: 0.0811 - val_accuracy: 0.9809
Epoch 9/20
180/180 [==============================] - 3s 14ms/step - loss: 0.0901 - accuracy: 0.9744 - val_loss: 0.0689 - val_accuracy: 0.9825
Epoch 10/20
180/180 [==============================] - 4s 21ms/step - loss: 0.0846 - accuracy: 0.9761 - val_loss: 0.0719 - val_accuracy: 0.9820
Epoch 11/20
180/180 [==============================] - 5s 25ms/step - loss: 0.0776 - accuracy: 0.9788 - val_loss: 0.0679 - val_accuracy: 0.9809
Epoch 12/20
180/180 [==============================] - 3s 15ms/step - loss: 0.0705 - accuracy: 0.9804 - val_loss: 0.0548 - val_accuracy: 0.9883
Epoch 13/20
180/180 [==============================] - 3s 14ms/step - loss: 0.0687 - accuracy: 0.9797 - val_loss: 0.0527 - val_accuracy: 0.9894
Epoch 14/20
180/180 [==============================] - 3s 15ms/step - loss: 0.0628 - accuracy: 0.9826 - val_loss: 0.0438 - val_accuracy: 0.9892
Epoch 15/20
180/180 [==============================] - 3s 14ms/step - loss: 0.0604 - accuracy: 0.9828 - val_loss: 0.0515 - val_accuracy: 0.9874
Epoch 16/20
180/180 [==============================] - 3s 14ms/step - loss: 0.0601 - accuracy: 0.9829 - val_loss: 0.0464 - val_accuracy: 0.9879
Epoch 17/20
180/180 [==============================] - 3s 14ms/step - loss: 0.0527 - accuracy: 0.9853 - val_loss: 0.0418 - val_accuracy: 0.9899
Epoch 18/20
180/180 [==============================] - 3s 14ms/step - loss: 0.0534 - accuracy: 0.9841 - val_loss: 0.0347 - val_accuracy: 0.9919
Epoch 19/20
180/180 [==============================] - 3s 14ms/step - loss: 0.0527 - accuracy: 0.9833 - val_loss: 0.0330 - val_accuracy: 0.9921
Epoch 20/20
180/180 [==============================] - 3s 14ms/step - loss: 0.0476 - accuracy: 0.9860 - val_loss: 0.0306 - val_accuracy: 0.9917

Wow, validation accuracy one again exceeds training accuracy, this time consistently hitting around 99%!

Third Model

For this last model we’re going to combine both text and title and will require us to concatinate our prior two pipelines:

main = layers.concatenate([title_features, text_features], axis=1)

Then add a final dense layer:

main = layers.Dense(32, activation='relu')(main)

Finally a new output layer:

output = layers.Dense(2, name="fake")(main) 

From here, it’s the same as before:

model3 = keras.Model(
    inputs = [title_input, text_input],
    outputs = output
)

model3.summary()

Model: "model_7"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to                     
==================================================================================================
 title (InputLayer)             [(None, 1)]          0           []                               
                                                                                                  
 text (InputLayer)              [(None, 1)]          0           []                               
                                                                                                  
 text_vectorization (TextVector  (None, 500)         0           ['title[0][0]',                  
 ization)                                                         'text[0][0]']                   
                                                                                                  
 embedding (Embedding)          (None, 500, 3)       6000        ['text_vectorization[0][0]']     
                                                                                                  
 embedding2 (Embedding)         (None, 500, 3)       6000        ['text_vectorization[2][0]']     
                                                                                                  
 dropout (Dropout)              (None, 500, 3)       0           ['embedding[0][0]']              
                                                                                                  
 dropout_4 (Dropout)            (None, 500, 3)       0           ['embedding2[0][0]']             
                                                                                                  
 global_average_pooling1d (Glob  (None, 3)           0           ['dropout[0][0]']                
 alAveragePooling1D)                                                                              
                                                                                                  
 global_average_pooling1d_2 (Gl  (None, 3)           0           ['dropout_4[0][0]']              
 obalAveragePooling1D)                                                                            
                                                                                                  
 dropout_1 (Dropout)            (None, 3)            0           ['global_average_pooling1d[0][0]'
                                                                 ]                                
                                                                                                  
 dropout_5 (Dropout)            (None, 3)            0           ['global_average_pooling1d_2[0][0
                                                                 ]']                              
                                                                                                  
 dense (Dense)                  (None, 32)           128         ['dropout_1[0][0]']              
                                                                                                  
 dense_3 (Dense)                (None, 32)           128         ['dropout_5[0][0]']              
                                                                                                  
 concatenate_1 (Concatenate)    (None, 64)           0           ['dense[0][0]',                  
                                                                  'dense_3[0][0]']                
                                                                                                  
 dense_4 (Dense)                (None, 32)           2080        ['concatenate_1[0][0]']          
                                                                                                  
 fake (Dense)                   (None, 2)            66          ['dense_4[0][0]']                
                                                                                                  
==================================================================================================
Total params: 14,402
Trainable params: 14,402
Non-trainable params: 0
__________________________________________________________________________________________________

keras.utils.plot_model(model3)

model3.compile(optimizer="adam",
              loss = losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=["accuracy"])

history3 = model3.fit(train, 
                    validation_data=val,
                    epochs = 20)

Epoch 1/20
180/180 [==============================] - 4s 17ms/step - loss: 0.1411 - accuracy: 0.9852 - val_loss: 0.0182 - val_accuracy: 0.9993
Epoch 2/20
180/180 [==============================] - 3s 16ms/step - loss: 0.0196 - accuracy: 0.9964 - val_loss: 0.0079 - val_accuracy: 0.9996
Epoch 3/20
180/180 [==============================] - 3s 16ms/step - loss: 0.0131 - accuracy: 0.9966 - val_loss: 0.0046 - val_accuracy: 0.9996
Epoch 4/20
180/180 [==============================] - 3s 17ms/step - loss: 0.0118 - accuracy: 0.9967 - val_loss: 0.0059 - val_accuracy: 0.9991
Epoch 5/20
180/180 [==============================] - 3s 16ms/step - loss: 0.0095 - accuracy: 0.9974 - val_loss: 0.0052 - val_accuracy: 0.9996
Epoch 6/20
180/180 [==============================] - 3s 16ms/step - loss: 0.0104 - accuracy: 0.9969 - val_loss: 0.0044 - val_accuracy: 0.9996
Epoch 7/20
180/180 [==============================] - 3s 16ms/step - loss: 0.0110 - accuracy: 0.9967 - val_loss: 0.0054 - val_accuracy: 0.9993
Epoch 8/20
180/180 [==============================] - 3s 16ms/step - loss: 0.0095 - accuracy: 0.9972 - val_loss: 0.0031 - val_accuracy: 0.9996
Epoch 9/20
180/180 [==============================] - 3s 16ms/step - loss: 0.0070 - accuracy: 0.9978 - val_loss: 0.0019 - val_accuracy: 0.9998
Epoch 10/20
180/180 [==============================] - 3s 16ms/step - loss: 0.0086 - accuracy: 0.9972 - val_loss: 0.0019 - val_accuracy: 0.9996
Epoch 11/20
180/180 [==============================] - 3s 16ms/step - loss: 0.0094 - accuracy: 0.9972 - val_loss: 0.0076 - val_accuracy: 0.9989
Epoch 12/20
180/180 [==============================] - 3s 16ms/step - loss: 0.0075 - accuracy: 0.9980 - val_loss: 0.0042 - val_accuracy: 0.9993
Epoch 13/20
180/180 [==============================] - 3s 16ms/step - loss: 0.0085 - accuracy: 0.9972 - val_loss: 0.0030 - val_accuracy: 0.9996
Epoch 14/20
180/180 [==============================] - 3s 16ms/step - loss: 0.0084 - accuracy: 0.9970 - val_loss: 8.1322e-04 - val_accuracy: 1.0000
Epoch 15/20
180/180 [==============================] - 3s 16ms/step - loss: 0.0072 - accuracy: 0.9978 - val_loss: 0.0020 - val_accuracy: 0.9991
Epoch 16/20
180/180 [==============================] - 3s 16ms/step - loss: 0.0082 - accuracy: 0.9974 - val_loss: 8.0624e-04 - val_accuracy: 0.9998
Epoch 17/20
180/180 [==============================] - 3s 16ms/step - loss: 0.0088 - accuracy: 0.9974 - val_loss: 0.0060 - val_accuracy: 0.9989
Epoch 18/20
180/180 [==============================] - 3s 16ms/step - loss: 0.0075 - accuracy: 0.9978 - val_loss: 0.0037 - val_accuracy: 0.9996
Epoch 19/20
180/180 [==============================] - 3s 16ms/step - loss: 0.0087 - accuracy: 0.9971 - val_loss: 5.9027e-04 - val_accuracy: 1.0000
Epoch 20/20
180/180 [==============================] - 3s 16ms/step - loss: 0.0059 - accuracy: 0.9984 - val_loss: 0.0028 - val_accuracy: 0.9996

The combined model achieves nearly perfect accuracy (over 99%). It fluctates between perfect, and a very fractional percentage off perfect, fairly quickly. You could likely get away with reducing the epochs for this model. Overall, good results.

Model Evaluation

Now let’s see how our best model (the main, combined model) performs on unseen data.

First let’s read in the test data from a url:

test_url = "https://github.com/PhilChodrow/PIC16b/blob/master/datasets/fake_news_test.csv?raw=true"
test_df = pd.read_csv(test_url)
test_df

	Unnamed: 0	title	text	fake
0	420	CNN And MSNBC Destroy Trump, Black Out His Fa...	Donald Trump practically does something to cri...	1
1	14902	Exclusive: Kremlin tells companies to deliver ...	The Kremlin wants good news. The Russian lead...	0
2	322	Golden State Warriors Coach Just WRECKED Trum...	On Saturday, the man we re forced to call Pre...	1
3	16108	Putin opens monument to Stalin's victims, diss...	President Vladimir Putin inaugurated a monumen...	0
4	10304	BREAKING: DNC HACKER FIRED For Bank Fraud…Blam...	Apparently breaking the law and scamming the g...	1
...	...	...	...	...
22444	20058	U.S. will stand be steadfast ally to Britain a...	The United States will stand by Britain as it ...	0
22445	21104	Trump rebukes South Korea after North Korean b...	U.S. President Donald Trump admonished South K...	0
22446	2842	New rule requires U.S. banks to allow consumer...	U.S. banks and credit card companies could be ...	0
22447	22298	US Middle Class Still Suffering from Rockefell...	Dick Eastman The Truth HoundWhen Henry Kissin...	1
22448	333	Scaramucci TV Appearance Goes Off The Rails A...	The most infamous characters from Donald Trump...	1

22449 rows × 4 columns

Now to run it through our existing preprocessing functions:

test_ds = make_dataset(test_df)

To test it:

model3.evaluate(test_ds)

225/225 [==============================] - 5s 21ms/step - loss: 0.0446 - accuracy: 0.9900

[0.04461757093667984, 0.9899773001670837]

Great, it scored 99% accuracy on the unseen dataset.

#Embedding Visualization

Finally, let’s make an embedding visualization to examine patterns and associations in the words the model found the most useful in this classification task.

weights = model3.get_layer('embedding').get_weights()[0] # get the weights from the embedding layer
vocab = vectorize_layer.get_vocabulary()                # get the vocabulary from our data prep for later

from sklearn.decomposition import PCA
pca = PCA(n_components=2)
weights = pca.fit_transform(weights)

embedding_df = pd.DataFrame({
    'word' : vocab, 
    'x0'   : weights[:,0],
    'x1'   : weights[:,1]
})

fig = px.scatter(embedding_df, 
                 x = "x0", 
                 y = "x1", 
                 size = [2]*len(embedding_df),
                # size_max = 2,
                 hover_name = "word")

fig.show()

So, looking at the visualization based on the weights extracted from our embedding layer what can what see?

“the” is on it…which really should be a stopword…and it has a hefty weight assigned to it. So what happened? Well, NLTK stop words as returned by “stopwords.words(‘english’)” are lowercase, so by removing stop words prior to standardization we’re missing the upper case instances. I know this assignment was inherited from Dr. Chodrow so, Shruti and Harlin, I know you won’t take offense, but the assignment dictating we handle stopwords prior to standardization seems like mistake? As a result, “The [insert thing or people you don’t like here] Problem….” “The [XZY] Threat” sort of inflamatory, hyper-partisan titles are included as is…so our assignment’s directed order of operations here has a flaw, that ironically actually preserved a meaningful pattern of misinformation headline. Happy accidents, right Bob Ross?
Politician’s first names are weighted far heavier than their lastnames. Anecdotally this makes sense to me: Trump’s pejorative called his 2016 opponent “crooked Hillary,” not “crooked Hillary Clinton.” It’s easier to implicitly and sexistly diminish her by refering to her as “Hillary”—emphasizing gender—rather than by her literally presidental last name. On the flip side, the most overtly biased commentators aren’t fond of saying “President Trump”, instead going with “Donald Trump”. It sticks in their mouth much in the way “President Obama” did for sensationalist right wing writers, who instead tended to use “Barack Obama” or “Barack Hussein Obama” to emphasize the “foreigness” of his name.
“Our”, “your”, and “america” are all clustered together. Once again, makes sense, nothing says click-bait like a personal appeal: “coming for your guns,” “ruining our country,” etc. It applies ownership to a larger concept and makes the sensationalized headline a personal matter.
Things people/groups never call themselves: “is” is next to “racist”, “leftist” is next to “radical”. I’m guessing the AP (Associated Press) wire didn’t flatout lead with “[insert candidate you don’t like here] is racist” or “radical leftists are ruining [insert something you like here]”. It’s just not how actual news phrases things.
Proper nouns for international people/places aren’t very weighted. “Japan”, “Macron”, “Myanmar” aren’t fodder for misinformation the way domestic matters seem to be. Once again, makes sense: foreign affairs aren’t salient and emotionally enough to the average American to rile them up to suspend disbelief, embrace confirmation bias, and eat up misinformation. In terms of the usefulness of these words for our classification, well, for every cry of “Trump is owned by the Kremlin” or “Biden is owned by China” theres probably threefold as many articles actually addressing actual trade policy with the world’s largest export manufacturer or Russia’s ongoing military actions in Syria or Ukraine…the latter two nearing a decade old. This obviously dilutes the usefulness of these words for our purposes.

Bonus observation:

“sources” and “says” are both words misinformation tends not to contain. I guess proper quotations and attributions aren’t a staple of the genre—”the prime minister’s presss office says [stuff],” “sources close to the matter…” that sort of thing. I’m also guessing that this applies to the “sources:” list at the bottom of proper articles.

Closing Thoughts

The model performed very well, however it appears to be very temporally sensitive. It is a useful tool for extrapolation within the same cultural/news epoch as its training data. That is to say that you could manually (at least as far as the fake/not fake classifcation) build a large training set and then automate the review of a larger body of articles from roughly the same period, however, the reliance of the model on specific phrasing/terminology rather than language structure means that it would likely perform far far worse with articles from 10 years ago or 10 years from now…and that’s being optimistic about it’s valid range of applicability. As always, thank you for coming to my TED talk.