SvelteKit and background workers

[hamboomger]

Hello there 👋
So, I've been using the SvelteKit for my own projects for some time, and for my last use case, I stumbled on a following question:

What is the "proper" way to execute .ts scripts in a sveltekit project?

Think of something like "management commands" concept in Django applications:

• The .ts script/command can be executed without starting up the server;
• You should be able to import any other module in this script, including imports like "$env/static/private".

Example:

Starting background worker for processing the message queue:

src/lib/commands/bg_worker.ts:

import { REDIS_HOST, REDIS_PORT } from "$env/static/private";
import { ConnectionOptions, Worker, Job } from "bullmq";
import { domainsRepository } from '$lib/server/DomainsRepository';

async function main() {
  const connection: ConnectionOptions = {
    host: REDIS_HOST,
    port: REDIS_PORT
  }
  new Worker('crawl-domain', async (job: Job) => {
    const { domainId } = job.data
    const domain = domainsRepository.getDomainById(domainId)
    // scrape domain
  }, { connection });
}

main()

package.json:

{
  "scripts": {
    "start_bg_worker": "... some way to run the bg_worker file"
  }
}

npm run start_bg_worker

Think about any other type of background jobs you might have to do on your backend. Workarounds and hacks are welcomed, but I'm mostly interested in finding out what is the "proper", SvelteKit way of doing it.

Update:

All thanks to @david-plugge:
#9807 (comment). vite-node solved it for me.

[0x2E]

#8794 (comment)

[hamboomger]

@0x2E thank you for the reply! The custom server is probably one option to solve it, but it doesn't really allow to do the second thing I mentioned:

• You should be able to import any other module in this script, including imports like "$env/static/private".

Let's say you want to use a DomainsRepository module in the script that starts the worker (which is a Typescript file):

// somewhere in the src/ directory

import { domainsRepository } from '$lib/server/DomainsRepository';  // <-- how would you import this?
import { REDIS_HOST, REDIS_PORT } from "$env/static/private";
import { ConnectionOptions, Worker, Job } from "bullmq";

async function main() {
  const connection: ConnectionOptions = {
    // ...options
  }
  new Worker('crawl-domain', async (job: Job) => {
    const { domainId } = job.data
    const domain = domainsRepository.getDomainById(domainId)
    // Crawl the domain
    return 'done';
  }, { connection });
}

For now the only way I see it is extracting the server logic into a separate npm workspace, and then using it as npm package in the SvelteKit app and in the background workers (which would be separate app as well). But the Django way to do this just seems so much easier to use.

[hamboomger]

I updated the original question, so that it's less vague.

[hamboomger]

No updates on this one?

[hamboomger]

@david-plugge nice, thank you! I had to split up my application into a standalone express.js server and frontend with Sveltekit, but the amount of things I have to take care of after the split is frankly overwhelming. I'll try vite-node instead and write an update later.

[hamboomger]

@0x2E admin interface? Not sure how would it help with long-running tasks (>60 seconds) on the server.

[0x2E]

I don't start and wait for the task. The task is running in the background and managed by bullmq.

BTW, I think it's better to split the "real" backend jobs into a separate service. The meta framework like SvelteKit is mainly designed for a simple request-response type backend.

[hamboomger]

@david-plugge I tried vite-node, works like a charm. I wish it was part of the SvelteKit documentation. Thank you, sir ❤️
@0x2E have you tried vite-node?

[0x2E]

Will give it a try.