Notes on semantic search over 60,000 papers
Building a Ctrl+F for science with OpenAlex, embeddings and FAISS.
Placeholder draft; replace with the real post.
The corpus came from the OpenAlex API: 60,000 abstracts across eight fields, about 9.45 million words, cleaned and stored as Parquet. The search is an embedding index in FAISS; the interesting problems were everywhere else.
Chunking is the product
Nobody searches for a whole abstract. The quality of results tracked almost perfectly with how the text was split (by sentence, by window, by section) long before the choice of embedding model mattered.
Evaluation before features
A tiny labelled set of fifty queries caught every regression that "looked fine" in manual testing. Build the harness first.