Typesense does't look like more suitable for our usecase. here why

[kcubeterm]

it is fast and have lots of features, but primary index location is RAM that means it uses same amount of ram the database size. Imagine you have 10k documents and after crawling it will surely output at least 100 MB data.

There's not much use-case of being fast in terms of Achoz. A few milliseconds does't make big difference. Also user does't search their data frequently so typesense occupies RAM resources which is not good for system performance.

we must look for a search engine which uses disk as index location. meilisearch do the same, but need to test some features as meilisearch comes with few limitations.

[cauerego]

i agree using ram and too much speed optimization goes against achoz goals.

we do want to index much more data than the user will have available as ram: at very least the entire storage of a cellphone! which typically will be at least 10x the amount of ram physically available.

according to typesense own documentation we would need, however 2x to 3x the amount of ram as the size of the dataset. completely different use case indeed.

and i also agree indexing will be an essential part of the project. we want to search through emails and even web content from 3rd parties, by scraping the user's web. but all this can come later...

i find it fundamental to weight in what technically sets us apart from basically all alternatives: agpl.

whenever we take code from any other license, we risk to lose access to it on the long run. because every code needs constant updates, even if very minor. and the maintainers of any code that didn't bother to understand why gnu and the fsf build the heart and brains of today's computing will eventually either do it and upgrade or get closed down by some money making company down the road.

in this aspect, moving to meilisearch feels like a huge step back.

of course, when comparing our goals and roadmap to any alternative, we can easily see what really sets us apart...

but i find it very important to take care of all details on the present.

above all, we need to move forward and get stuff done, and working, so as long as you, my friends, can find motivation to keep going, please, do it as you will.

memory, speed, license, all this becomes secondary.

too long (or confusing); didn't read (or understood);

if we can't use ripgrepall (agpl) or combine it with some other such cli (anything gnu will be at least gpl+), i don't know of any better alternative today.

please, go ahead with meilisearch, if that moves you! 😘

[kcubeterm]

but need to test some features as meilisearch comes with few limitations.

Quite slow indexing. Only reason I cant go with it. we will stick with typesense until we find better option. Do you thing mongodb (agpl) work like search engine for us, it would be tricky offcourse.

[cauerego]

i think it would be too tricky to use any database system as a "search engine".

building a good search is no easy task.

stallman himself features a terrible search on his website! 🤣

[kcubeterm]

In order to make achoz cross platform we can use both search engine. it would be user choice to use which search engine they want to use.
I will be more happy to add more search engine compatibility if someone request for it,

[cauerego]

for now, let's get it working exactly as expected in linux first, for the demo. just 1 engine. it's a lot of with as it is.

[kcubeterm]

well. I tested meilisearch with over 300 documents, all were pdf with more than 50k words. Indexing in batch is fine and viable for achoz. and also search result was not that fast but viable in case of achoz. the query i tries, average time it took was around 500ms. It would be slow in case of millions of documents,but still viable and average performer. and I am going for it. The default search engine would be meilisearch, and the user who need better performance, could always opt typesense.

[cauerego]

yes, half a second to get results, given the alternatives, is plenty enough.

what's more, even if searx would always take 1 second (which is 10 times worse than duck), but offered good enough results, i would never use duck again. i already never use google.

that's too say that 1 second might sound like too much when comparing raw data results... but for a user experience on "searching through everything", and as a beta software, man, that's perfectly fine! 😘