Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rsc: Update the tool to fully bootstrap a fresh db from one command #1546

Merged
merged 3 commits into from
Apr 25, 2024
Merged

rsc: Update the tool to fully bootstrap a fresh db from one command #1546

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c179394
rsc: Update the tool to fully bootstrap a fresh db from one command
V-FEXrt Apr 23, 2024
51fc46b
fix test
V-FEXrt Apr 23, 2024
1e3e38d
address comments
V-FEXrt Apr 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions rust/rsc/src/common/database.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,38 @@ pub async fn delete_local_blob_store(
blob_store::Entity::delete_by_id(store.id).exec(db).await
}

// --------------------------------------------------
// ---------- DbOnly Blob Store ----------
// --------------------------------------------------
// ---------- Create ----------
pub async fn create_dbonly_blob_store(db: &DatabaseConnection) -> Result<blob_store::Model, DbErr> {
let active_blob_store = blob_store::ActiveModel {
id: Set(read_dbonly_blob_store_id()),
r#type: Set("DbOnlyBlobStore".to_string()),
};

active_blob_store.insert(db).await
}

// ---------- Read ----------
// This creates a single source of truth for the DbOnly Blob Store id in case it ever needs to
// change. Unwrap is safe here since the string being parsed is static.
pub fn read_dbonly_blob_store_id() -> Uuid {
Uuid::parse_str("00000000-0000-0000-0000-000000000000").unwrap()
}

pub async fn read_dbonly_blob_store(
db: &DatabaseConnection,
) -> Result<Option<blob_store::Model>, DbErr> {
blob_store::Entity::find_by_id(read_dbonly_blob_store_id())
.one(db)
.await
}

// ---------- Update ----------

// ---------- Delete ----------

// --------------------------------------------------
// ---------- Job ----------
// --------------------------------------------------
Expand Down
20 changes: 16 additions & 4 deletions rust/rsc/src/rsc/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ mod types;

#[path = "../common/config.rs"]
mod config;
#[path = "../common/database.rs"]
mod database;

#[derive(Debug, Parser)]
struct ServerOptions {
Expand Down Expand Up @@ -101,10 +103,18 @@ async fn activate_stores(
}

// --- Activate DBOnly Store ---
let dbonly_id = Uuid::parse_str("00000000-0000-0000-0000-000000000000").unwrap();
let dbonly_store = match database::read_dbonly_blob_store(&conn).await {
Ok(Some(store)) => store,
Ok(None) => {
panic!("Database is not configured with a DbOnly store. Please bootstrap the db")
}
Err(_) => panic!("Failed to load DbOnly store from database. Unable to continue"),
};
active_stores.insert(
dbonly_id,
Arc::new(blob_store_impls::DbOnlyBlobStore { id: dbonly_id }),
dbonly_store.id,
Arc::new(blob_store_impls::DbOnlyBlobStore {
id: dbonly_store.id,
}),
);

return active_stores;
Expand All @@ -128,7 +138,7 @@ fn create_router(
panic!("UUID for active store not in database");
};

let dbonly_uuid = Uuid::parse_str("00000000-0000-0000-0000-000000000000").unwrap();
let dbonly_uuid = database::read_dbonly_blob_store_id();
let Some(dbonly_store) = blob_stores.get(&dbonly_uuid).clone() else {
panic!("UUID for db store not in database");
};
Expand Down Expand Up @@ -407,6 +417,8 @@ mod tests {
async fn create_test_store(
db: &DatabaseConnection,
) -> Result<Uuid, Box<dyn std::error::Error>> {
database::create_dbonly_blob_store(db).await?;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm surprised we don't have to update some check now that the database has two active stores.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding the dbonly store does muddy the terminology a bit, but there is still only one "active" store.

The active store is just the store that, by default, a blob is pushed to. The dbonly store is always enabled but not the default.

Happy to use a different term if you have any suggestions

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a test that this default dbonly blob store is working properly?

Regarding naming it seems like active store makes sense from the perspective of someone pushing blobs, but from someone pulling blobs they may get results from "inactive" stores. default also doesn't solve this confusion. It seems like "push" or "write" or some verb like that would need to be joined with the other adjective, so "default_push_store" or "active_write_store" or things like this but I'm not sure any of those combinations actually provide clarity, and may impair understanding.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a test that this default dbonly blob store is working properly?

No not yet, I'll get one added in a followup. Testing the blob uploads is a bit more tricky since I need to stream files but it should be done


let test_store = blob_store::ActiveModel {
id: NotSet,
r#type: Set("TestBlobStore".into()),
Expand Down
61 changes: 50 additions & 11 deletions rust/rsc/src/rsc_tool/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
use clap::{Parser, Subcommand};
use inquire::Confirm;
use inquire::{Confirm, Text};
use is_terminal::IsTerminal;
use migration::{DbErr, Migrator, MigratorTrait};
use sea_orm::{prelude::Uuid, DatabaseConnection};
Expand All @@ -14,11 +14,14 @@ mod config;
mod database;

async fn add_api_key(
opts: AddApiKeyOpts,
key: Option<String>,
desc: String,
db: &DatabaseConnection,
) -> Result<(), Box<dyn std::error::Error>> {
let key = database::create_api_key(db, opts.key, opts.desc).await?;
println!("Created api key: {}", key.id);
let key = database::create_api_key(db, key, desc).await?;
println!("Created api key:");
println!(" Id: {}", key.id);
println!(" Key: {}", key.key);
Ok(())
}

Expand Down Expand Up @@ -74,12 +77,13 @@ async fn remove_api_key(
}

async fn add_local_blob_store(
opts: AddLocalBlobStoreOpts,
root: String,
db: &DatabaseConnection,
) -> Result<(), Box<dyn std::error::Error>> {
tokio::fs::create_dir_all(opts.root.clone()).await?;
let store = database::create_local_blob_store(db, opts.root).await?;
println!("Created local blob store: {}", store.id);
tokio::fs::create_dir_all(root.clone()).await?;
let store = database::create_local_blob_store(db, root).await?;
println!("Created local blob store:");
println!(" Id: {}", store.id);
Ok(())
}

Expand Down Expand Up @@ -166,6 +170,33 @@ async fn remove_all_jobs(db: &DatabaseConnection) -> Result<(), Box<dyn std::err
Ok(())
}

async fn bootstrap_db(db: &DatabaseConnection) -> Result<(), Box<dyn std::error::Error>> {
// Get all the info from the user
let use_dev_key = Confirm::new("Create a test API key? (development only!)")
.with_default(false)
.prompt()?;
let key_desc = Text::new("What is the key used for?").prompt()?;
let local_store_root = Text::new("Where should local blobs be saved?").prompt()?;

// Create the API key
let key = if use_dev_key {
Some("InsecureKey".into())
} else {
None
};
add_api_key(key, key_desc, db).await?;

// Create the local blob store
add_local_blob_store(local_store_root, db).await?;

// Create DbOnly blob store
database::create_dbonly_blob_store(db).await?;

println!("");
println!("Done! Use the store id to set the active store and share the key as appropiate.");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting the active store is something that user does via another rsc_tool invocation?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its a part of the launch parameters into the RSC. You aren't allowed to change the active store while the RSC is running since the "activation" of a store is a non trivial action on a nearly global hashmap

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should probably add a README to rsc_tool to cover common use patterns that aren't covered by --help

Ok(())
}

// Define all of our top level commands
#[derive(Debug, Parser)]
#[command(author, version, about, long_about = None)]
Expand All @@ -185,9 +216,12 @@ struct TopLevel {
)]
database_url: Option<String>,

#[arg(help = "Show's the config and then exits", long)]
#[arg(help = "Shows the config and then exits", long)]
show_config: bool,

#[arg(help = "Bootstraps the database with minimal configs", long)]
bootstrap: bool,

#[command(subcommand)]
db_command: Option<DBCommand>,
}
Expand Down Expand Up @@ -326,6 +360,11 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
Err(err)?;
}

if args.bootstrap {
bootstrap_db(&db).await?;
return Ok(());
}

let Some(db_command) = args.db_command else {
return Ok(());
};
Expand All @@ -342,10 +381,10 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
// Add Commands
DBCommand::Add(AddOpts {
db_command: DBModelAdd::ApiKey(args),
}) => add_api_key(args, &db).await?,
}) => add_api_key(args.key, args.desc, &db).await?,
DBCommand::Add(AddOpts {
db_command: DBModelAdd::LocalBlobStore(args),
}) => add_local_blob_store(args, &db).await?,
}) => add_local_blob_store(args.root, &db).await?,

// Remove Commands
DBCommand::Remove(RemoveOpts {
Expand Down
Loading