Search: optimize embeddings chunks

[josephjclark]

The chunks in the search service are currently, so far as I can tell, sub optimal.

Looking over the chunks manually, many are unhelpful. The granularity isn't quite right.

I'd also like to see metadata attached to chunks so that we can even report that metadata back to users (what resources did the AI use to answer your question?)

Basically we just need a pass of validation and quality for the main docs and adaptor chunks.