MG-2287 - Improve Search For Things

[nyagamunene]

What type of PR is this?

This is a feature because it improve search for Things

What does this do?

This enables back-end support for filtering by name, ID and tags.

Which issue(s) does this PR fix/relate to?

• Related Issue Improve search for Users, Things, Channels, and Groups #2287
• Resolves Improve search for Users, Things, Channels, and Groups #2287

Have you included tests for your changes?

Yes, I have included tests for my changes.

Did you document any new/modified feature?

No. I have not updated the documentation because it will be done in a different PR

Notes

[arvindh123]

I think this change is not required, since we have SearchBasicInfo function in things postgres

[nyagamunene]

From the comment left by @dborovcanin here, I understood that the renaming is necessary to avoid confusion between retrieve and search.

[arvindh123]

Yeah, I agree
But the RetrieveAllBasicInfo will be no more needed , it will be replaced by search.

[nyagamunene]

Okay then

[arvindh123]

I think this change is not required, since we have SearchBasicInfo function in things postgres

[dborovcanin]

Avoid importing API to repo even if it's internal.

[dborovcanin]

This change does not belong to this PR.

[dborovcanin]

There is no need to use regular expresions, since we do not have an advanced query. LIKE or ILIKE should be a better fit and faster.

[arvindh123]

@nyagamunene

Instead of SearchBasicInfo can rename to SearchClients ?

[nyagamunene]

We can rename it to SearchClients no problem

[arvindh123]

Why here we are calling SearchThing function
And not calling repo function svc.clients.SearchBasicInfo(ctx, pm)
`

[nyagamunene]

In the SearchThings were are checking if the user has authority to view the thing so it is separated from listclients when retrieving the basic info

[arvindh123]

Authorization are done on above switch case.

[arvindh123]

Authorization are done on above switch case.

[arvindh123]

This will not return proper page items.

[arvindh123]

I think we don't need this SearchThings function in service layer

[nyagamunene]

I will remove the SearchThings function then.

[nyagamunene]

The authorization is needed because the member should not be able to view things created by other members

[arvindh123]

Yeah, In ListThings it is handled.

[arvindh123]

Yeah, In ListThings it is handled.

[arvindh123]

Something like this will make code reusable and make simpler.

Suggested change

	}
	func buildQueryConditions(pm clients.Page) ([]string) {
	var query []string
	if pm.Name != "" {
	query = append(query, "name ILIKE '%' || :name || '%'")
	}
	if pm.Identity != "" {
	query = append(query, "identity ILIKE '%' || :identity || '%'")
	}
	if pm.Id != "" {
	query = append(query, "id ILIKE '%' || :id || '%'")
	}
	if pm.Tag != "" {
	query = append(query, "EXISTS (SELECT 1 FROM unnest(tags) AS tag WHERE tag ILIKE '%' || :tag || '%')")
	}
	if len(pm.IDs) != 0 {
	query = append(query, fmt.Sprintf("id IN ('%s')", strings.Join(pm.IDs, "','")))
	}
	return query
	}
	func applyOrdering(emq string, pm clients.Page) string {
	switch pm.Order {
	case "name", "identity", "created_at", "updated_at":
	emq = fmt.Sprintf("%s ORDER BY %s", emq, pm.Order)
	if pm.Dir == api.AscDir || pm.Dir == api.DescDir {
	emq = fmt.Sprintf("%s %s", emq, pm.Dir)
	}
	}
	return emq
	}
	func PageQuery(pm clients.Page) (string, error) {
	mq, _, err := postgres.CreateMetadataQuery("", pm.Metadata)
	if err != nil {
	return "", errors.Wrap(errors.ErrMalformedEntity, err)
	}
	query := buildQueryConditions(pm)
	if mq != "" {
	query = append(query, mq)
	}
	// Additional conditions specific to PageQuery
	if pm.Status != clients.AllStatus {
	query = append(query, "c.status = :status")
	}
	if pm.Domain != "" {
	query = append(query, "c.domain_id = :domain_id")
	}
	if pm.Role != clients.AllRole {
	query = append(query, "c.role = :role")
	}
	if len(query) > 0 {
	emq = fmt.Sprintf("WHERE %s", strings.Join(query, " AND "))
	}
	var emq string
	if len(query) > 0 {
	emq = fmt.Sprintf("WHERE %s", strings.Join(query, " AND "))
	}
	emq = applyOrdering(emq, pm)
	return emq, nil
	}
	func ConstructSearchQuery(pm clients.Page) (string) {
	query := buildQueryConditions(pm)
	var emq string
	if len(query) > 0 {
	emq = fmt.Sprintf("WHERE %s", strings.Join(query, " AND "))
	}
	emq = applyOrdering(emq, pm)
	return emq, tq
	}

[dborovcanin]

Why is this test removed?

[dborovcanin]

This function is used only once. Remove it and put the code directly to the SearchClients.

[nyagamunene]

Okay

[dborovcanin]

We are treating IDs differently, i.e. taking it into account only if other criteria is not there, making buildQueryConditions mask the logic of processing IDs. Why?

[nyagamunene]

This is allows searchClients to retrieve basic info by id or search. So if either name, tag, identify, or id is set we don't retrieve by id

[dborovcanin]

Why don't we still combine? If you want only IDs, you send only IDs in the query.

[nyagamunene]

We can but to get you properly, you are suggesting we combine both searchClients and retrieveAllByIds right?

[dborovcanin]

Eventually, that would be the outcome here, yes. I.e. Search users from this pool of IDs with this criteria. It's unlikely to need such a thing, but it makes buildQueryConditions consistent.

[arvindh123]

@dborovcanin @nyagamunene
For search with ID, it will work even the ID matches partially.
So we need list of things to which user have access.
As you know, Things service could not have information of user accessiable thing, so it will retreive from spiceDB for all the IDs to which user have access.

Once we find way to list things efficently with access control, then we can remove this IDs from entities

[arvindh123]

@dborovcanin @nyagamunene
For search with ID, it will work even the ID matches partially.
So we need list of things to which user have access.
As you know, Things service could not have information of user accessiable thing, so it will retreive from spiceDB for all the IDs to which user have access.

Once we find way to list things efficently with access control, then we can remove this IDs from entities

[arvindh123]

Why do we need to pulicate the mgclients.Page, instead we can pass variable pm directly to SearchClients

Suggested change

	pg, err := svc.clients.SearchClients(ctx, p)
	pg, err := svc.clients.SearchClients(ctx, pm)

[dborovcanin]

Remove this function and streamline PageQuery to something like:

Suggested change

	func buildSearch(query []string, pm clients.Page) string {
	func PageQuery(pm clients.Page) (string, error) {
	mq, _, err := postgres.CreateMetadataQuery("", pm.Metadata)
	if err != nil {
	return "", errors.Wrap(errors.ErrMalformedEntity, err)
	}
	var query []string
	if pm.Name != "" {
	query = append(query, "name ILIKE '%' || :name || '%'")
	}
	if pm.Identity != "" {
	query = append(query, "identity ILIKE '%' || :identity || '%'")
	}
	if pm.Id != "" {
	query = append(query, "id ILIKE '%' || :id || '%'")
	}
	if pm.Tag != "" {
	query = append(query, "EXISTS (SELECT 1 FROM unnest(tags) AS tag WHERE tag ILIKE '%' || :tag || '%')")
	}
	if pm.Role != clients.AllRole {
	query = append(query, "c.role = :role")
	}
	// If there are search params presents, use search and ignore other options.
	// Always combine role with search params, so len(query) > 1.
	if len(query) > 1 {
	return fmt.Sprintf("WHERE %s", strings.Join(query, " AND ")), nil
	}
	if mq != "" {
	query = append(query, mq)
	}
	if len(pm.IDs) != 0 {
	query = append(query, fmt.Sprintf("id IN ('%s')", strings.Join(pm.IDs, "','")))
	}
	if pm.Status != clients.AllStatus {
	query = append(query, "c.status = :status")
	}
	if pm.Domain != "" {
	query = append(query, "c.domain_id = :domain_id")
	}
	var emq string
	if len(query) > 0 {
	emq = fmt.Sprintf("WHERE %s", strings.Join(query, " AND "))
	}
	return emq, nil
	}

Don't forget comments, it's important because the function is a little cryptic. We must improve, I have opened a ticket for this: #2326.