Binary Quantization from Scratch
Setup: Install Dependencies, Imports & Download Embeddings
!pip install matplotlib tqdm pandas numpy datasets --quiet --upgrade
import numpy as np
import pandas as pd
from datasets import load_dataset
from tqdm import tqdm
👨🏾💻 Code Walkthrough
Here's an explanation of the code structure provided:
- Loading Data: OpenAI embeddings are loaded from a parquet files (we can load upto 1M embedding) and concatenated into one array.
- Binary Conversion: A new array with the same shape is initialized with zeros, and the positive values in the original vectors are set to 1.
- Accuracy Function: The accuracy function compares original vectors with binary vectors for a given index, limit, and oversampling rate. The comparison is done using dot products and logical XOR, sorting the results, and measuring the intersection.
- Testing: The accuracy is tested for different oversampling rates (1, 2, 4), revealing a correctness of ~0.96 for an oversampling of 4.
💿 Loading Data
# Download from Huggingface Hub
ds = load_dataset(
"Qdrant/dbpedia-entities-openai3-text-embedding-3-large-3072-100K", split="train"
)
openai_vectors = np.array(ds["text-embedding-3-large-3072-embedding"])
del ds
openai_bin = np.zeros_like(openai_vectors, dtype=np.int8)
openai_bin[openai_vectors > 0] = 1
n_dim = openai_vectors.shape[1]
n_dim
🎯 Accuracy Function
We will use the accuracy function to compare the original vectors with the binary vectors for a given index, limit, and oversampling rate. The comparison is done using dot products and logical XOR, sorting the results, and measuring the intersection.
def accuracy(idx, limit: int, oversampling: int):
scores = np.dot(openai_vectors, openai_vectors[idx])
dot_results = np.argsort(scores)[-limit:][::-1]
bin_scores = n_dim - np.logical_xor(openai_bin, openai_bin[idx]).sum(axis=1)
bin_results = np.argsort(bin_scores)[-(limit * oversampling) :][::-1]
return len(set(dot_results).intersection(set(bin_results))) / limit
📊 Results
number_of_samples = 10
limits = [3, 10]
sampling_rate = [1, 2, 3, 5]
results = []
def mean_accuracy(number_of_samples, limit, sampling_rate):
return np.mean(
[accuracy(i, limit=limit, oversampling=sampling_rate) for i in range(number_of_samples)]
)
for i in tqdm(sampling_rate):
for j in tqdm(limits):
result = {
"sampling_rate": i,
"limit": j,
"mean_acc": mean_accuracy(number_of_samples, j, i),
}
print(result)
results.append(result)
㆓ Binary Conversion
Here, we will use 0 as the threshold for the binary conversion. All values greater than 0 will be set to 1, and others will remain 0. This is a simple and effective way to convert continuous values into binary values for OpenAI embeddings.
results = pd.DataFrame(results)
results