Building Machine Learning Systems with a Feature Store: Batch, Real-Time, and LLM Systems cover
Building Machine Learning Systems with a Feature Store: Batch, Real-Time, and LLM Systems
by Jim Dowling
ISBN: 1098165233
Found in 1 comment on Hacker News
View on Amazon
We may earn a commission from purchases made through links on this page.
Not ready yet? Get weekly book picks.
jamesblonde · 2026-04-01 · Original thread
I say this quite a lot to data scientists who are now building agents:

1. think of the context data as training data for your requests (the LLM performs in-context learning based on your provided context data)

2. think of evals as test data to evaluate the performance of your agents. Collect them from agent traces and label them manually. If you want to "train" a LLM to act as a judge to label traces, then again, you will need lots of good quality examples (training data) as the LLM-as-a-Judge does in-context learning as well.

From my book - https://www.amazon.com/Building-Machine-Learning-Systems-Fea...