Things I got wrong building an incremental indexing pipeline [P](reddit.com)

I've been working on incremental indexing pipelines lately, basically keeping a vector store in sync as the source data changes, and I keep finding the same bugs never show up until it's been running a while. Biggest one for me is deletes. I tested the "new doc comes in, gets embedded" path a hundred times and it was fine. Never really tested what happens when a doc gets deleted upstream. Turns out if you don't handle that, your index just keeps growing with stuff that shouldn't be there anymore, and you don't notice until search starts returning weird results. Partial updates got me too. I didn't want to re-embed a whole doc every time something small changed so I did partial updates instead. Cheaper, but I ended up with drift between what's in the index and what's actually true in the source, especially once chunk boundaries moved around. Didn't notice until a query happened to hit the stale part. Also learned the hard way that idempotency isn't optional. My pipeline gets retried and backfilled all the time, and if reprocessing the same input twice doesn't give the same result, I get duplicate docs every time something routine reruns. None of this feels like new information, it's just normal distributed systems stuff, but I feel like it gets way less discussion than embedding models or chunking strategies. Anyone else dealt with this or have a setup that's actually held up long term? submitted by /u/Whole-Assignment6240 [link] [Kommentare]