What is the recommended/fastest way to search for tools in the Tool Shed API?

[davelopez]

I'm working on supporting tool_id search/auto-completion for Format2 workflow files.

The idea is to roughly type the identifier (or maybe the name would be better 🤔) of a tool and then invoke intelli-sense by pressing Ctrl+Space. This will search the Tool Shed for the top 5 tools that match the ID and let you select the desired one to auto-complete with the full signature.

Here is a proof of concept:

Unfortunately, the time to search for tools is too long to be usable 😞
The screencast above has been cut while "loading..." to a couple of seconds, but it actually took more than a minute to perform the search 😓

Here is a snippet to show how I'm doing the search:

searchToolshedTools.js

const toolshedUrl = "https://toolshed.g2.bx.psu.edu";

async function searchToolshedTools(query, maxResults = 5) {
  const url = `${toolshedUrl}/api/tools?q=${query}&page_size=${maxResults}`;
  console.log(`Fetching: ${url}`);

  const startTime = Date.now();
  const response = await fetch(url);
  const endTime = Date.now();

  console.log(`Time taken: ${endTime - startTime}ms`);

  const json = await response.json();
  console.log("Response: ", JSON.stringify(json, null, 2));
}

const whooshQuery = "id:diff";
searchToolshedTools(whooshQuery);

If I run this with node searchToolshedTools.js I get something like this:

Show full output

Fetching: https://toolshed.g2.bx.psu.edu/api/tools?q=id:diff&page_size=5
Time taken: 93049ms
Response:  {
  "total_results": "15",
  "page": "1",
  "page_size": "5",
  "hits": [
    {
      "tool": {
        "id": "diffacto",
        "repo_owner_username": "galaxyp",
        "repo_name": "diffacto",
        "name": "Diffacto",
        "description": "Comparative Protein Abundance from Covariation of Peptide Abundances"
      },
      "matched_terms": {
        "id": "diffacto"
      },
      "score": 13.903021514393922
    },
    {
      "tool": {
        "id": "difftrans",
        "repo_owner_username": "vimalkumarvelayudhan",
        "repo_name": "riboseqr_wrapper",
        "name": "Differential Translation Analysis",
        "description": "\n        (Step 4) Get Ribo and RNA-Seq counts with riboSeqR. Perform differential\n        translation analysis with baySeq.\n    "
      },
      "matched_terms": {
        "id": "difftrans"
      },
      "score": 13.903021514393922
    },
    {
      "tool": {
        "id": "diffexp",
        "repo_owner_username": "shians",
        "repo_name": "voom_rnaseq",
        "name": "Voom Rnaseq",
        "description": "\n      Perform differential expression analysis using pipeline based on the voom\n      function of the limma bioconductor package. This tool takes a count matrix\n      (tab separated) as input and produces a HTML report as output.\n  "
      },
      "matched_terms": {
        "id": "diffexp"
      },
      "score": 13.607996689504494
    },
    {
      "tool": {
        "id": "diffbind",
        "repo_owner_username": "bgruening",
        "repo_name": "diffbind",
        "name": "DiffBind",
        "description": " differential binding analysis of ChIP-Seq peak data"
      },
      "matched_terms": {
        "id": "diffbind"
      },
      "score": 13.607996689504494
    },
    {
      "tool": {
        "id": "diff",
        "repo_owner_username": "bgruening",
        "repo_name": "diff",
        "name": "diff",
        "description": "analyzes two files and generates an unidiff text file with information about the differences and an optional Html report"
      },
      "matched_terms": {
        "id": "diff"
      },
      "score": 13.607996689504494
    }
  ],
  "hostname": "https://toolshed.g2.bx.psu.edu/"
}

Fetching: https://toolshed.g2.bx.psu.edu/api/tools?q=id:diff&page_size=5
Time taken: 93049ms

A couple of seconds is fine, but waiting 93 (sometimes more) seconds for the response seems unreasonable 😮‍💨

My questions are:

• Am I using the wrong API endpoint, query, or complete approach?

• If not, how can we improve the search speed?

I'm not very familiar yet with the Tool Shed codebase, but I guess there is a good reason this takes so much time.
I'm happy to work on a solution if there is anything we can do to improve that! I'd love to read your ideas.

Thanks! 🙏

[mvdbeek]

FWIW the query is faster if you drop the id: prefix, making me think something is just wrong with the whoosh search. That said even that query takes about 6 seconds.
Optimizing whoosh is not a pleasant thing to do, and it took us many years to get a usable whoosh search on the Galaxy side (I was in favor of just removing it altogether).

I do think that's the endpoint to use, and maybe we can improve the search but I'm a little pessimistic in that area. I think we should have focused on a simple, exact search before we went into more complicated searches.

[davelopez]

Thank you Marius for taking the time to look into it!

FWIW the query is faster if you drop the id: prefix

Yeah, I was surprised by that too, however, the results are way less accurate. A simple query for diff will match 1490 tools instead of the 15 for id:diff and the exact match tool I want to use diff will appear in the 8th place instead of 1st 🫤

Those improvements you mention seem to make a lot of sense. I wonder if we could try them in the Test Tool Shed to see if they make a significant difference. If they do and the search goes back to a couple of seconds that would be awesome.

The same query to the Test Tool Shed gives similar times for the id: search:

Fetching: https://toolshed.g2.bx.psu.edu/api/tools?q=id:diff&page_size=5
Time taken: 111446ms

In case this doesn't improve much the situation, will it make sense to add a DB table with all the tools and a handful of fields like (id, name, description, latest_version, repo_owner_username, repo_name) and use it for simple searches like this or would it be a terrible naive idea? I guess the bigger issue here would be ensuring the integrity and synchronization of tools 🤔

[mvdbeek]

Yeah, we can try them, but maybe I can just grab the existing whoosh index and test it out locally 😆.

In case this doesn't improve much the situation, will it make sense to add a DB table with all the tools and a handful of fields like (id, name, description, latest_version, repo_owner_username, repo_name) and use it for simple searches like this or would it be a terrible naive idea?

I think this would make a lot of sense to me, and perhaps that table should also contain the precomputed schema instead of a disk-based cache. But lets check those relatively simple tweaks first.

[davelopez]

and perhaps that table should also contain the precomputed schema instead of a disk-based cache.

I like that idea a lot. I wonder what @jmchilton thinks about it 😃

[jmchilton]

I had started moving in this direction in my tool shed branch. That table would open so many doors. The problem is simply that it is hard. It is much deeper than the artificial changes I’ve mostly been making to the tool shed. We’ve got update those table on every commit - which is easy enough. But then when we recalculate metadata for a repository - they might need to be updated - versions can get dropped or added depending on what happens to installable revisions. The SQL Alchemy is pretty awkward also… like are we deleting these objects and recreating them on each metadata computation and are we creating objects for each revision and doing something in an third table or view to track which objects are the latest for each given tool version. Is all that happening mixed in with the existing code or afterward. What does the transaction-ing look like.

It is important work that needs to be done, but it is hard. It is why I wrote tests for metadata recalculating and put a lot of work into refactoring and adding types to that code. It also may be the case that Celery could really cause us to rethink all of that - but I am anxious to add to the admin burden of the tool shed - it doesn’t have an admin time it needs as is I think.

[davelopez]

I kind of imagined that synchronization for data integrity would be hard to achieve. But I'm happy we are already taking steps in that direction and is being considered. Thank you! 🙏

[hexylena]

I've had a repository that scrapes the toolshed and creates various metadata files, for a while since the TS doesn't give me what I need (though it will in the future, very excited to deprecate this!)

If you would want a single large json blob that had every tool id:description, it would be trivial to add that to our data exports. That would not get you search, but maybe the UX of "slow first query and instant subsequent ones" is fine?

[davelopez]

Thank you @hexylena! That is interesting! Maybe with a bit of pre-processing, I could use that as a fall-back offline solution 👍

[hexylena]

Ah it's already there, these files would have what you need, albeit with every version of the tool.

$ du -h tool-meta.*
3.6M	tool-meta.json
2.6M	tool-meta.tsv

in tool-meta.json:

{
  "toolshed.g2.bx.psu.edu/repos/imgteam/2d_auto_threshold/ip_threshold/0.0.3": {
    "name": "Auto Threshold",
    "desc": "applies a standard threshold algorithm to an image",
    "edam_operations": [
      "operation_3443",
      "operation_3553",
      "operation_0337",
      "operation_2409"
    ],
    "edam_topics": [
      "topic_3068",
      "topic_3382",
      "topic_3372"
    ]
  },
  "toolshed.g2.bx.psu.edu/repos/imgteam/2d_auto_threshold/ip_threshold/0.0.4": {
    "name": "Auto Threshold",
    "desc": "applies a standard threshold algorithm to an image",
    "edam_operations": [
      "operation_3443",
      "operation_0337",
      "operation_2409",
      "operation_3553"
    ],
    "edam_topics": [
      "topic_3068",
      "topic_3382",
      "topic_3372"
    ]
  },

dropping edam gets you down to 2.3M / 180K gzip'd

$ cat tool-meta.json | jq '[. | to_entries[] | {"key": .key, "value": (.value.name + " " +.value.desc)}] | from_entries' -S > tmp
$ du -h tmp
2.3M	tmp
$ gzip tmp
$ du -h tmp.gz 
180K	tmp.gz

which could be marginally smaller by removing non-latest versions, though, I think this would get you pretty close to a reasonable number? That's not a bad size.

edit: now available. https://hexylena.github.io/toolshed-version-database/tool-meta-names.json