Anomaly detection with embeddings

View on Generative AI Run in Google Colab View source on GitHub


This tutorial demonstrates how to use the embeddings from the PaLM API to detect potential outliers in your dataset. You will visualize a subset of the 20 Newsgroup dataset using t-SNE and detect outliers outside a particular radius of the central point of each categorical cluster.

For more information on getting started with embeddings generated from the PaLM API, check out the quickstart.


First, download and install the PaLM API Python library.

pip install -q google-generativeai
import google.generativeai as palm

import re
import tqdm
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.datasets import fetch_20newsgroups
from sklearn.manifold import TSNE

Grab an API Key

To get started, you'll need to create an API key.

models = [m for m in palm.list_models() if 'embedText' in m.supported_generation_methods]

model = models[0]

Prepare dataset

The 20 Newsgroups Text Dataset contains 18,000 newsgroups posts on 20 topics divided into training and test sets. The split between the training and test datasets are based on messages posted before and after a specific date. This tutorial uses the training subset.

newsgroups_train = fetch_20newsgroups(subset='train')

# View list of class names for dataset

Here is the first example in the training set.

idx =[0].index('Lines')
Lines: 15

 I was wondering if anyone out there could enlighten me on this car I saw
the other day. It was a 2-door sports car, looked to be from the late 60s/
early 70s. It was called a Bricklin. The doors were really small. In addition,
the front bumper was separate from the rest of the body. This is 
all I know. If anyone can tellme a model name, engine specs, years
of production, where this car is made, history, or whatever info you
have on this funky looking car, please e-mail.


- IL
   ---- brought to you by your neighborhood Lerxst ----
# Apply functions to remove names, emails, and extraneous words from data points in = [re.sub(r'[\w\.-]+@[\w\.-]+', '', d) for d in] # Remove email = [re.sub(r"\([^()]*\)", "", d) for d in] # Remove names = [d.replace("From: ", "") for d in] # Remove "From: " = [d.replace("\nSubject: ", "") for d in] # Remove "\nSubject: "
# Put training points into a dataframe
df_train = pd.DataFrame(, columns=['Text'])
df_train['Label'] =
# Match label to target name index
df_train['Class Name'] = df_train['Label'].map(newsgroups_train.target_names.__getitem__)


Next, sample some of the data by taking 150 data points in the training dataset and choosing a few categories. This tutorial uses the science categories.

# Take a sample of each label category from df_train
df_train = (df_train.groupby('Label', as_index = False)
                    .apply(lambda x: x.sample(SAMPLE_SIZE))

# Choose categories about science
df_train = df_train[df_train['Class Name'].str.contains('sci')]

# Reset the index
df_train = df_train.reset_index()
df_train['Class Name'].value_counts()
sci.crypt          150
sci.electronics    150            150          150
Name: Class Name, dtype: int64

Create the embeddings

In this section, you will see how to generate embeddings for the different texts in the dataframe using the embeddings from the PaLM API.

from google.api_core import retry

# Define a function with automatic retry logic to compute embeddings and wait
# a bit once the quota limit has been reached.
def embed_text(model, text):
  x = palm.generate_embeddings(model=model, text=text)
  return np.array(x['embedding'])

# Create embeddings for each document and add that as a column to the dataframe
df_train['Embeddings'] = df_train['Text'].progress_apply(lambda x: embed_text(model=model,
df_train.drop('index', axis=1, inplace=True)
100%|██████████| 600/600 [04:10<00:00,  2.39it/s]

Dimensionality reduction

The dimension of the document embedding vector is 768. In order to visualize how the embedded documents are grouped together, you will need to apply dimensionality reduction as you can only visualize the embeddings in 2D or 3D space. Contextually similar documents should be closer together in space as opposed to documents that are not as similar.

# Convert df_train['Embeddings'] Pandas series to a np.array of float32
X = np.array(df_train['Embeddings'].to_list(), dtype=np.float32)
(600, 768)

You will apply the t-Distributed Stochastic Neighbor Embedding (t-SNE) to perform dimensionality reduction. This technique reduces the number of dimensions, while preserving clusters (points that are close together stay close together). For the original data, the model tries to construct a distribution over which other data points are "neighbors" (e.g., they share a similar meaning). It then optimizes an objective function to keep a similar distribution in the visualization.

tsne = TSNE(random_state=0, n_iter=1000)
tsne_results = tsne.fit_transform(X)
df_tsne = pd.DataFrame(tsne_results, columns=['TSNE1', 'TSNE2'])
df_tsne['Class Name'] = df_train['Class Name'] # Add labels column from df_train to df_tsne
fig, ax = plt.subplots(figsize=(8,6)) # Set figsize
sns.set_style('darkgrid', {"grid.color": ".6", "grid.linestyle": ":"})
sns.scatterplot(data=df_tsne, x='TSNE1', y='TSNE2', hue='Class Name', palette='Set2')
sns.move_legend(ax, "upper left", bbox_to_anchor=(1, 1))
plt.title('Scatter plot of news using t-SNE')


Outlier detection

To determine which points are anomalous, you will determine which points are inliers and outliers. Start by finding the centroid, or location that represents the center of the cluster, and use the distance to determine the points that are outliers.

Start by getting the centroid of each category.

def get_centroids(df_tsne):
  # Get the centroid of each cluster
  centroids = df_tsne.groupby('Class Name').mean()
  return centroids

centroids = get_centroids(df_tsne)
def get_embedding_centroids(df):
  emb_centroids = dict()
  grouped = df.groupby('Class Name')
  for c in grouped.groups:
    sub_df = grouped.get_group(c)
    # Get the centroid value of dimension 768
    emb_centroids[c] = np.mean(sub_df['Embeddings'], axis=0)

  return emb_centroids
emb_c = get_embedding_centroids(df_train)

Plot each centroid you have found against the rest of the points.

# Plot the centroids against the cluster
fig, ax = plt.subplots(figsize=(8,6)) # Set figsize
sns.set_style('darkgrid', {"grid.color": ".6", "grid.linestyle": ":"})
sns.scatterplot(data=df_tsne, x='TSNE1', y='TSNE2', hue='Class Name', palette='Set2');
sns.scatterplot(data=centroids, x='TSNE1', y='TSNE2', color="black", marker='X', s=100, label='Centroids')
sns.move_legend(ax, "upper left", bbox_to_anchor=(1, 1))
plt.title('Scatter plot of news using t-SNE with centroids')


Choose a radius. Anything beyond this bound from the centroid of that category is considered an outlier.

def calculate_euclidean_distance(p1, p2):
  return np.sqrt(np.sum(np.square(p1 - p2)))

def detect_outlier(df, emb_centroids, radius):
  for idx, row in df.iterrows():
    class_name = row['Class Name'] # Get class name of row
    # Compare centroid distances
    dist = calculate_euclidean_distance(row['Embeddings'],
                                        emb_centroids[class_name])[idx, 'Outlier'] = dist > radius

  return len(df[df['Outlier'] == True])
range_ = np.arange(0.3, 0.75, 0.02).round(decimals=2).tolist()
num_outliers = []
for i in range_:
  num_outliers.append(detect_outlier(df_train, emb_c, i))
# Plot range_ and num_outliers
fig = plt.figure(figsize = (14, 8))
plt.rcParams.update({'font.size': 12}), range_)), num_outliers)
plt.title("Number of outliers vs. distance of points from centroid")
plt.ylabel("Number of outliers")
for i in range(len(range_)):
  plt.text(i, num_outliers[i], num_outliers[i], ha = 'center')


Depending on how sensitive you want your anomaly detector to be, you can choose which radius you would like to use. For now, 0.58 is used, but you can change this value.

# View the points that are outliers
RADIUS = 0.62
detect_outlier(df_train, emb_c, RADIUS)
df_outliers = df_train[df_train['Outlier'] == True]
# Use the index to map the outlier points back to the projected TSNE points
outliers_projected = df_tsne.loc[df_outliers['Outlier'].index]

Plot the outliers and denote them using a transparent red color.

fig, ax = plt.subplots(figsize=(8,6)) # Set figsize
plt.rcParams.update({'font.size': 10})
sns.set_style('darkgrid', {"grid.color": ".6", "grid.linestyle": ":"})
sns.scatterplot(data=df_tsne, x='TSNE1', y='TSNE2', hue='Class Name', palette='Set2');
sns.scatterplot(data=centroids, x='TSNE1', y='TSNE2', color="black", marker='X', s=100, label='Centroids')
# Draw a red circle around the outliers
sns.scatterplot(data=outliers_projected, x='TSNE1', y='TSNE2', color='red', marker='o', alpha=0.5, s=90, label='Outliers')
sns.move_legend(ax, "upper left", bbox_to_anchor=(1, 1))
plt.title('Scatter plot of news with outliers projected with t-SNE')


Use the index values of the datafames to print a few examples of what outliers can look like in each category. Here, the first data point from each category is printed out. Explore other points in each category to see data that are deemed as outliers, or anomalies.

sci_crypt_outliers = df_outliers[df_outliers['Class Name'] == 'sci.crypt']
Re: **Sorry folks** 
Organization: Enormes_Rebajas_Online
Distribution: na
Lines: 15

In article <>   writes:

>I just found out from my source that this article was a joke.  Heh heh..  
>It seemed pretty damn convincing to me from the start -- I just didn't
>notice the smiley at the end of the article, and there were a few other
>hints which I should of caught.

    People took this article seriously?  I mean, I know it's the
Net and all, but the prankster didn't even have Clinton's sound-bites

Rita Rouvalis
sci_elec_outliers = df_outliers[df_outliers['Class Name'] == 'sci.electronics']
Re: Does someone know what is the news group for IEEE.
Distribution: usa
Organization: SFBAC
Lines: 11
X-Newsreader: Helldiver 1.07 

In <>   writes:
>            Thanks a lot.




are the most frequently used groups.
sci_med_outliers = df_outliers[df_outliers['Class Name'] == '']
Re: Can men get yeast infections?
Lines: 13


 LB> I know from personal experience that men CAN get yeast infections. I 
 LB> get rather nasty ones from time to time, mostly in the area of the
 LB> scrotum and the base of the penis. 

I used to have problems with recurrent athlete's foot until I 
started drying between my toes with my blow drier after each time 
I bathe.  I also dry my pubic area while I am at it to prevent 
problems.  You might want to try it.

... My cat types with his tail.

 * Origin: ONE WORLD Los Angeles 310/372-0987 32b
sci_space_outliers = df_outliers[df_outliers['Class Name'] == '']
Stereo Pix of planets?
Organization: California State University, Sacramento
Lines: 5

Can anyone tell me where I might find stereo images of planetary and
planetary satellite surfaces?  GIFs preferred, but any will do.  I'm
especially interested in stereos of the surfaces of Phobos, Deimos, Mars
and the Moon .

Next steps

You've now created an anomaly detector using embeddings! Try using your own textual data to visualize them as embeddings, and choose some bound such that you can detect outliers. You can perform dimensionality reduction in order to complete the visualization step. Note that TSNE is good at clustering inputs, but can take a longer time to converge or might get stuck at local minima. If you run into this issue, another technique you could consider are principal components analysis (PCA).

To learn more about how you can use the embeddings, check out the examples available. To learn how to create them from scratch, see TensorFlow's Word Embeddings tutorial. To learn how to use other services in the PaLM API, visit the various quickstart guides: