Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* dockerizing filetao; Add envars to 'docs/docker/run-filetao-with-docker.md' * dockerizing filetao; Use full path for miner data volume * dockerizing filetao; Allowing to use non-default port for Redis * dockerizing filetao; removing unnecessary volume * Fixed wallet creation in user registration. * dockerizing filetao; improving redis container for non-default port * dockerizing filetao; improving redis container for non-default port * updating 'check_environment' function and adapting code * Wallet generation fixes. * remove unused code, add mnemonic * add redis db, singleton server wallet * Added Procfile, gitignore and runtime.txt for launching app on Heroku. * Fix dependency conflict in requirements.txt * Fix additional dependency conflict in requirements.txt * Fix fastapi bittensor dependency conflict in requirements.txt * Fix fastapi uvicorn dependency conflict in requirements.txt * FastAPI slug too large. Updated webdev gitignore to match main repo. * FastAPI slug too large. Added api, cli, miner, shared, validator, tests to gitignore. * Added django frontend & set everything up * support py311 * add scripts and dereg script to docker image * return str from get_block_seed error * Added FileResponse to retrieve_data() for client file download. * update reamde for 3.11 support * update names from database * fix import from storage for API * install torc-cpu first so we override bittensor req to use cuda * update to test and import fixes in main, add hotkeys for db * set min priority for hotkeys not registered to SN21 * add metagraph check for updating cached * fix bug in get_reids_size rm await * add convenience data getters for validator * add tier statistics calc * add tmp scratch * remove save check, no longer turned on by default * Updated requirements.txt for heroku slug build. Installing torch cpu before bittensor. * Fixed package version conflict between bittensor and fastapi. * Fixed package version conflict between bittensor and uvicorn. * Added PYTHONPATH=/app to Procfile. Fix for storage import. * Updated requirements.txt with filetao. Prep for heroku deployment. * Pushed Procfile to top level. * Fixed entry point in Procfile. * Fixed database import in main.py * Added redis env vars to database.py * Updated env var REDIS_DB to string * Updated env var REDIS_PORT to string * Updated env var REDIS_HOST * Updated get_database with Heroku provided REDIS_URL * update database, fix improper scan_iter usage and add totalreqs * bugfix and add inverted dict for tier stats * add wandb project link to readme * return proper dict, add compute by tier stats * add updated sql queries * add testnet blurb * fix sql command to add data to network table * Added bcrypt library to requirements.txt to fix generate_seed() in main.py * Update database.py * Added env var ACCESS_TOKEN_EXPIRE_MINUTES to main.py * Changed env var ACCESS_TOKEN_EXPIRE_MINUTES type to str * Added ALGORITHM to env var in main.py * version 3 * use get for missing values * fix perms * fix perms * version bump * handle case of no successful responses * update API to store hotkey metadata and pull from when retrieving * check for uid in monitor lookup * add hotkeys missing arg in retrieve * add missing imports * add pwd context crypt * except when mnemonic fails to generate * open access for priority also * return hotkeys * get proper return from upload, use test.txt filename * get metagraph to fetch hotkeys from metadata * fix api retrieve priority for open access * fix retrieve * fix test for retrieve * print new return values * readme blurb for docker * add getters for dict obj in tier stats comp * use x- alias instead of dummy service in docker-compose.yml * dont build custom redis docker image * remove dummy filetao service * remove -prod postfix and container_name from docker-compose to make it easier to setup multiple instances * optimize dockerfile & docker-compose.yml config * docker-compose fixes, more ENV options, and make some truely optional * use python3.11 in docker for extra speed boost * modify metadata schema to be more useful, add endpoints * convert metadata schema, add convenience endpoints for metadata * Added changes to include redis password * use imported bittensor * update API to return first successful parallel query * detour dendrite for more accurate time * drop REDIS_CONF env var as there is no need for it anymore * further simplify dockerized setup * Begin indexer for api neuron * Make api folder a module * Fix imports * Fix import * Rename api helpers to indexer * Fix endpoint import * Fix import one more time for indexer.endpoint * set proc time after processing synapse * Fix majority of issues in indexer * refactor redis to separate file * Sqlite dont check for thread safety * Wait before collecting data in indexer, refactor endpoint to pull from database * Remove unused code * (WIP) add delete to subnet, add delet endpoint to backend * return 400 for file already exists with store * Added CORS to Fast API. Also made changes to accommodate Redis password. Added env.example * Fix timing with neuron creation and indexer thread start, store incentive in HotkeysTable * update excluse list properly in ping uids * http 422 error on file exists in upload_file * decrease variance slightly on sigm normalization * replace sub handler hotkey * readme updates * low-level api demo notebook * fix metadata overwrite issue, mirror dropbox behavior * import fix rename_file and delete_cid_meta * import DeleteUser * cancel extra tasks in api to not linger * add retry mechanism to api * correct typos * mark all miner errors with non-200 status codes * (WIP) expanded tier set * allow skipping Redis checks * fix type annotation breaking under py39 * Update readme to reflect rewards, storage constants * Update main.py Update generate cid args to use raw data bytes. * Update main.py Add bool for whether or not to increment file count when uploading (or overwriting) * Update database.py Add bool for incrementing file count in metdata func * Update main.py Actually still incr * redis pw warning if not found, don't exit * error -> warning for redis pw * add config to redis init * pass on non-registered miners (old) * skip attempt to index if no cap data found * Update config.py * Update base.py * Update store.py * Update retrieve.py * Update store_api.py * Update retrieve_api.py * Update challenge.py * redis pw warning if not found, don't exit * error -> warning for redis pw * update fastapi req * use torch cpu if desired in instructions * add readme for backend * typo fix * change -> filename_exists() * remove webdev reqs, update pycryptodome * remove test txt * remove comment * blurb about extend use * remove bittensor type hint for lint * update bittensor version and indexer import struct * add loguru now that bittensor>611 doesn't use it * typo * need pandas * need bigger than BIGINT for storage value * convert network cap and usage to mb for BIGINT fitting * convert all storage usage/cap values to MB * typo * ensure data directory is created * max chunk size 512 -> 256 * use mainnet finney * update DELETE methods * refactor data cap * udpate readme to new storage caps * readme fix for tier advance calc * constants fix, adjust wilson * avoid div/0 err * default to finney * Added check to see if user is over storage limit when uploading file. * bespoke blacklist for delete * proper synapse * Fix max user storage in backend * Fix argparse argument definition for api.whitelisted_hotkeys * nargs fix2 * allow whitelisted priority * Fix deletion functionality in backend * Fix delete_user_data not passing user wallet to delete api * Fix user wallet retrieval in delete file api * select max 3 valis at a time in backend * Remove axons/uids argument from delete api call * Fix import error for random library * Iterate over list in delete_user_data and check encryption * Add debug print to delete_user_data metadata * Change print func for debug to bt logging * Start axon even if serving fails * Disable call to deleting file on miner * speed up indexer and remove non-registered hotkeys from consideration * convert cap and usage to MB in tier calc * structure output * fix delete payload * reenable delete in backend * proper update UserInDB storage value * Remove debug print outputting all data from retrieved file * cleanup delete api * fix tier stats to use MB instead of bytes * Bugfix/cli testing fixes (#186) * Removed await for function returning None * Argparse moved to outside async main to prevent unhandled exception from polluting stdout * Reverted missing await for run command. Need to investigate where Nonetype is being produced. * Some formatting updates to docker section for readme * add uid(s) arg for store/retrieve * Add indexer endpoint for hotkey-specific data * Fix double function name definition * Fix get_specific_miner_stats route * Remove apirouter definition * Make start_time, end_time optional args with defaults of an hour, add offset argument * Return empty array if query returns None * Fix order by addition to sqlite queries * fix store priority for whitelist * remove print * default whitelist key update * Add total subnet emission to network stats table * Add frontend whitelist back to default whitelisted hotkeys * Change out wrong whitelisted hotkey * add reamde to frontend * Fix wrong hotkey in default whitelist * Add hotkey emission to indexer database * Add emission to indexer endpoint api * removing the delete from storage at the moment. Taking way too long to process. --------- Co-authored-by: eduardo <[email protected]> Co-authored-by: Joshua Johnson <[email protected]> Co-authored-by: Saevitus0 <[email protected]> Co-authored-by: Maciej Urbanski <[email protected]> Co-authored-by: peterbcarlson41 <[email protected]> Co-authored-by: Ayden Brewer <[email protected]>
- Loading branch information
7 people
authored
Apr 30, 2024
1 parent
b92052e
commit def46f8