Welcome to url-frontier Discussions!

[jnioche]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[Chaiavi]

Hi Julien, thanks for adding me.

If I may start with a question which I am wondering about.

I saw the draft you posted, and it seems like the frontier will be accessed using an external API using gRPC calls.

Our current CC components are used as an internal library with minimal third party dependencies (Do we still use even Tika?), the url-fronier sounds like a heavier one.

What is your suggestion - architecture-wise
Please note that some crawlers (the small ones) might prefer having it as an internal library
Maybe have the API of it in CC and the implementation as an implementation suggestion ?

[jnioche]

Hi Avi

Hi Julien, thanks for adding me.

I think all CC committers are added to this project automatically. Thanks for chiming in, glad you are interested!

I saw the draft you posted, and it seems like the frontier will be accessed using an external API using gRPC calls. Our current CC components are used as an internal library with minimal third party dependencies (Do we still use even Tika?), the url-fronier sounds like a heavier one.

That's why it is in a separate project from our library stuff in crawler-commons. Despite being a bit different in nature from our other projects, it is in the same spirit of sharing & reusing.

Please note that some crawlers (the small ones) might prefer having it as an internal library
Maybe have the API of it in CC and the implementation as an implementation suggestion ?

what we are aiming for is service-based, this is not negotiable ;-) in practice, the implementation(s) will be Docker-based and a crawler, no matter how small, could use the API to connect to it. What I am hoping is that there will also be implementations running as a service and commercial offerings so that users will have the choice as to where they go for the URLFrontier without having to rewrite a single line of code (hopefully). So in a way, the implementation we will provide here is as you put it, just a suggestion or starting point.

Note that we will probably have a simple, single process memory-based implementation. This of course could be either running as a service within the same JVM as a small Java crawler or used directly as a dependency. Even though it won't be scalable, it will be useful initially to define the API and have something we can play with.

[jnioche]

Hi @Chaiavi

Apparently you can use an In-process transport mode . Haven't used it though