To create a fleet, create a YAML file in your project folder. Its name must end with .dstack.yml (e.g. .dstack.yml or fleet.dstack.yml
diff --git a/search/search_index.json b/search/search_index.json
index c7df8d0c..9ab5ea74 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"],"fields":{"title":{"boost":1000.0},"text":{"boost":1.0},"tags":{"boost":1000000.0}}},"docs":[{"location":"terms/","title":"Terms of service","text":""},{"location":"terms/#agreement-to-terms","title":"Agreement to terms","text":"
We are dstack GmbH (\"Company,\" \"we,\" \"us,\" \"our\"), a company registered in Germany at Franz-Joseph-Stra\u00dfe, 11, Munich, Bayern 80801.
These Legal Terms constitute a legally binding agreement made between you, whether personally or on behalf of an entity (\"you\"), and dstack GmbH, concerning your access to and use of the Services. You agree that by accessing the Services, you have read, understood, and agreed to be bound by all of these Legal Terms. IF YOU DO NOT AGREE WITH ALL OF THESE LEGAL TERMS, THEN YOU ARE EXPRESSLY PROHIBITED FROM USING THE SERVICES AND YOU MUST DISCONTINUE USE IMMEDIATELY.
Supplemental terms and conditions or documents that may be posted on the Services from time to time are hereby expressly incorporated herein by reference. We reserve the right, in our sole discretion, to make changes or modifications to these Legal Terms from time to time. We will alert you about any changes by updating the \"Last updated\" date of these Legal Terms, and you waive any right to receive specific notice of each such change. It is your responsibility to periodically review these Legal Terms to stay informed of updates. You will be subject to, and will be deemed to have been made aware of and to have accepted, the changes in any revised Legal Terms by your continued use of the Services after the date such revised Legal Terms are posted.
"},{"location":"terms/#1-our-services","title":"1. Our services","text":"The information provided when using the Services is not intended for distribution to or use by any person or entity in any jurisdiction or country where such distribution or use would be contrary to law or regulation or which would subject us to any registration requirement within such jurisdiction or country. Accordingly, those persons who choose to access the Services from other locations do so on their own initiative and are solely responsible for compliance with local laws, if and to the extent local laws are applicable.
The Services are not tailored to comply with industry-specific regulations (Health Insurance Portability and Accountability Act (HIPAA), Federal Information Security Management Act (FISMA), etc.), so if your interactions would be subjected to such laws, you may not use the Services. You may not use the Services in a way that would violate the Gramm-Leach-Bliley Act (GLBA).
"},{"location":"terms/#2-intelliectual-property-rights","title":"2. Intelliectual property rights","text":"Our intellectual property
We are the owner or the licensee of all intellectual property rights in our Services, including all source code, databases, functionality, software, website designs, audio, video, text, photographs, and graphics in the Services ( collectively, the \"Content\"), as well as the trademarks, service marks, and logos contained therein (the \"Marks\").
Our Content and Marks are protected by copyright and trademark laws (and various other intellectual property rights and unfair competition laws) and treaties in the United States and around the world.
The Content and Marks are provided in or through the Services \"AS IS\" for your personal, non-commercial use or internal business purpose only.
Your use of our Services
Subject to your compliance with these Legal Terms, including the \"Prohibited activities\" section below, we grant you a non-exclusive, non-transferable, revocable license to:
Except as set out in this section or elsewhere in our Legal Terms, no part of the Services and no Content or Marks may be copied, reproduced, aggregated, republished, uploaded, posted, publicly displayed, encoded, translated, transmitted, distributed, sold, licensed, or otherwise exploited for any commercial purpose whatsoever, without our express prior written permission.
If you wish to make any use of the Services, Content, or Marks other than as set out in this section or elsewhere in our Legal Terms, please address your request to: hello@dstack.ai. If we ever grant you the permission to post, reproduce, or publicly display any part of our Services or Content, you must identify us as the owners or licensors of the Services, Content, or Marks and ensure that any copyright or proprietary notice appears or is visible on posting, reproducing, or displaying our Content.
We reserve all rights not expressly granted to you in and to the Services, Content, and Marks.
Any breach of these Intellectual Property Rights will constitute a material breach of our Legal Terms and your right to use our Services will terminate immediately.
Your submissions
Please review this section and the \"Prohibited activities\" section carefully prior to using our Services to understand the (a) rights you give us and (b) obligations you have when you post or upload any content through the Services.
Submissions: By directly sending us any question, comment, suggestion, idea, feedback, or other information about the Services (\"Submissions\"), you agree to assign to us all intellectual property rights in such Submission. You agree that we shall own this Submission and be entitled to its unrestricted use and dissemination for any lawful purpose, commercial or otherwise, without acknowledgment or compensation to you.
You are responsible for what you post or upload: By sending us Submissions through any part of the Services you: * confirm that you have read and agree with our \"Prohibited activities\" and will not post, send, publish, upload, or * transmit through the Services any Submission that is illegal, harassing, hateful, harmful, defamatory, obscene, * bullying, abusive, discriminatory, threatening to any person or group, sexually explicit, false, inaccurate, deceitful, or misleading; * to the extent permissible by applicable law, waive any and all moral rights to any such Submission; * warrant that any such Submission are original to you or that you have the necessary rights and licenses to submit such Submissions and that you have full authority to grant us the above-mentioned rights in relation to your Submissions; and * warrant and represent that your Submissions do not constitute confidential information.
You are solely responsible for your Submissions and you expressly agree to reimburse us for any and all losses that we may suffer because of your breach of (a) this section, (b) any third party\u2019s intellectual property rights, or (c) applicable law.
"},{"location":"terms/#3-user-representations","title":"3. User representations","text":"By using the Services, you represent and warrant that: (1) all registration information you submit will be true, accurate, current, and complete; (2) you will maintain the accuracy of such information and promptly update such registration information as necessary; (3) you have the legal capacity and you agree to comply with these Legal Terms; ( 4) you are not a minor in the jurisdiction in which you reside; (5) you will not access the Services through automated or non-human means, whether through a bot, script or otherwise; (6) you will not use the Services for any illegal or unauthorized purpose; and (7) your use of the Services will not violate any applicable law or regulation.
If you provide any information that is untrue, inaccurate, not current, or incomplete, we have the right to suspend or terminate your account and refuse any and all current or future use of the Services (or any portion thereof).
"},{"location":"terms/#4-user-registration","title":"4. User registration","text":"You may be required to register to use the Services. You agree to keep your password confidential and will be responsible for all use of your account and password. We reserve the right to remove, reclaim, or change a username you select if we determine, in our sole discretion, that such username is inappropriate, obscene, or otherwise objectionable.
"},{"location":"terms/#5-purchases-and-payment","title":"5. Purchases and payment","text":"We accept the following forms of payment:
You agree to provide current, complete, and accurate purchase and account information for all purchases made via the Services. You further agree to promptly update account and payment information, including email address, payment method, and payment card expiration date, so that we can complete your transactions and contact you as needed. Sales tax will be added to the price of purchases as deemed required by us. We may change prices at any time. All payments shall be in US dollars.
You agree to pay all charges at the prices then in effect for your purchases and any applicable shipping fees, and you authorize us to charge your chosen payment provider for any such amounts upon placing your order. We reserve the right to correct any errors or mistakes in pricing, even if we have already requested or received payment.
We reserve the right to refuse any order placed through the Services. We may, in our sole discretion, limit or cancel quantities purchased per person, per household, or per order. These restrictions may include orders placed by or under the same customer account, the same payment method, and/or orders that use the same billing or shipping address. We reserve the right to limit or prohibit orders that, in our sole judgment, appear to be placed by dealers, resellers, or distributors.
"},{"location":"terms/#6-subscriptions","title":"6. Subscriptions","text":"Billing and Renewal
e.g. by topping up their balance manually using their credit card.
Cancellation
You can cancel your subscription at any time by contacting us using the contact information provided below. Your cancellation will take effect at the end of the current paid term. If you have any questions or are unsatisfied with our Services, please email us at hello@dstack.ai .
Fee Changes
We may, from time to time, make changes to the subscription fee and will communicate any price changes to you in accordance with applicable law.
"},{"location":"terms/#7-software","title":"7. Software","text":"We may include software for use in connection with our Services. If such software is accompanied by an end user license agreement (\"EULA\"), the terms of the EULA will govern your use of the software. If such software is not accompanied by a EULA, then we grant to you a non-exclusive, revocable, personal, and non-transferable license to use such software solely in connection with our services and in accordance with these Legal Terms. Any software and any related documentation is provided \"AS IS\" without warranty of any kind, either express or implied, including, without limitation, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. You accept any and all risk arising out of use or performance of any software. You may not reproduce or redistribute any software except in accordance with the EULA or these Legal Terms.
"},{"location":"terms/#8-prohibited-activities","title":"8. Prohibited activities","text":"You may not access or use the Services for any purpose other than that for which we make the Services available. The Services may not be used in connection with any commercial endeavors except those that are specifically endorsed or approved by us.
As a user of the Services, you agree not to:
The Services does not offer users to submit or post content.
"},{"location":"terms/#10-contribution-license","title":"10. Contribution license","text":"You and Services agree that we may access, store, process, and use any information and personal data that you provide following the terms of the Privacy Policy and your choices (including settings).
By submitting suggestions or other feedback regarding the Services, you agree that we can use and share such feedback for any purpose without compensation to you.
"},{"location":"terms/#11-social-media","title":"11. Social media","text":"As part of the functionality of the Services, you may link your account with online accounts you have with third-party service providers (each such account, a \"Third-Party Account\") by either: (1) providing your Third-Party Account login information through the Services; or (2) allowing us to access your Third-Party Account, as is permitted under the applicable terms and conditions that govern your use of each Third-Party Account. You represent and warrant that you are entitled to disclose your Third-Party Account login information to us and/or grant us access to your Third-Party Account, without breach by you of any of the terms and conditions that govern your use of the applicable Third-Party Account, and without obligating us to pay any fees or making us subject to any usage limitations imposed by the third-party service provider of the Third-Party Account. By granting us access to any Third-Party Accounts, you understand that (1) we may access, make available, and store (if applicable) any content that you have provided to and stored in your Third-Party Account (the \"Social Network Content\") so that it is available on and through the Services via your account, including without limitation any friend lists and (2) we may submit to and receive from your Third-Party Account additional information to the extent you are notified when you link your account with the Third-Party Account. Depending on the Third-Party Accounts you choose and subject to the privacy settings that you have set in such Third-Party Accounts, personally identifiable information that you post to your Third-Party Accounts may be available on and through your account on the Services. Please note that if a Third-Party Account or associated service becomes unavailable or our access to such Third-Party Account is terminated by the third-party service provider, then Social Network Content may no longer be available on and through the Services. You will have the ability to disable the connection between your account on the Services and your Third-Party Accounts at any time. PLEASE NOTE THAT YOUR RELATIONSHIP WITH THE THIRD-PARTY SERVICE PROVIDERS ASSOCIATED WITH YOUR THIRD-PARTY ACCOUNTS IS GOVERNED SOLELY BY YOUR AGREEMENT(S) WITH SUCH THIRD-PARTY SERVICE PROVIDERS. We make no effort to review any Social Network Content for any purpose, including but not limited to, for accuracy, legality, or non-infringement, and we are not responsible for any Social Network Content. You acknowledge and agree that we may access your email address book associated with a Third-Party Account and your contacts list stored on your mobile device or tablet computer solely for purposes of identifying and informing you of those contacts who have also registered to use the Services. You can deactivate the connection between the Services and your Third-Party Account by contacting us using the contact information below or through your account settings (if applicable). We will attempt to delete any information stored on our servers that was obtained through such Third-Party Account, except the username and profile picture that become associated with your account.
"},{"location":"terms/#12-third-party-websites-and-content","title":"12. Third-party websites and content","text":"The Services may contain (or you may be sent via the Site) links to other websites (\"Third-Party Websites\") as well as articles, photographs, text, graphics, pictures, designs, music, sound, video, information, applications, software, and other content or items belonging to or originating from third parties (\"Third-Party Content\"). Such Third-Party Websites and Third-Party Content are not investigated, monitored, or checked for accuracy, appropriateness, or completeness by us, and we are not responsible for any Third-Party Websites accessed through the Services or any Third-Party Content posted on, available through, or installed from the Services, including the content, accuracy, offensiveness, opinions, reliability, privacy practices, or other policies of or contained in the Third-Party Websites or the Third-Party Content. Inclusion of, linking to, or permitting the use or installation of any Third-Party Websites or any Third-Party Content does not imply approval or endorsement thereof by us. If you decide to leave the Services and access the Third-Party Websites or to use or install any Third-Party Content, you do so at your own risk, and you should be aware these Legal Terms no longer govern. You should review the applicable terms and policies, including privacy and data gathering practices, of any website to which you navigate from the Services or relating to any applications you use or install from the Services. Any purchases you make through Third-Party Websites will be through other websites and from other companies, and we take no responsibility whatsoever in relation to such purchases which are exclusively between you and the applicable third party. You agree and acknowledge that we do not endorse the products or services offered on Third-Party Websites and you shall hold us blameless from any harm caused by your purchase of such products or services. Additionally, you shall hold us blameless from any losses sustained by you or harm caused to you relating to or resulting in any way from any Third-Party Content or any contact with Third-Party Websites.
"},{"location":"terms/#13-services-management","title":"13. Services management","text":"We reserve the right, but not the obligation, to: (1) monitor the Services for violations of these Legal Terms; (2) take appropriate legal action against anyone who, in our sole discretion, violates the law or these Legal Terms, including without limitation, reporting such user to law enforcement authorities; (3) in our sole discretion and without limitation, refuse, restrict access to, limit the availability of, or disable (to the extent technologically feasible) any of your Contributions or any portion thereof; (4) in our sole discretion and without limitation, notice, or liability, to remove from the Services or otherwise disable all files and content that are excessive in size or are in any way burdensome to our systems; and (5) otherwise manage the Services in a manner designed to protect our rights and property and to facilitate the proper functioning of the Services.
"},{"location":"terms/#14-privacy-policy","title":"14. Privacy policy","text":"We care about data privacy and security. Please review our Privacy Policy. By using the Services, you agree to be bound by our Privacy Policy, which is incorporated into these Legal Terms. Please be advised the Services are hosted in Germany and United States. If you access the Services from any other region of the world with laws or other requirements governing personal data collection, use, or disclosure that differ from applicable laws in Germany and United States, then through your continued use of the Services, you are transferring your data to Germany and United States, and you expressly consent to have your data transferred to and processed in Germany and United States.
"},{"location":"terms/#15-term-and-termination","title":"15. Term and termination","text":"These Legal Terms shall remain in full force and effect while you use the Services. WITHOUT LIMITING ANY OTHER PROVISION OF THESE LEGAL TERMS, WE RESERVE THE RIGHT TO, IN OUR SOLE DISCRETION AND WITHOUT NOTICE OR LIABILITY, DENY ACCESS TO AND USE OF THE SERVICES (INCLUDING BLOCKING CERTAIN IP ADDRESSES), TO ANY PERSON FOR ANY REASON OR FOR NO REASON, INCLUDING WITHOUT LIMITATION FOR BREACH OF ANY REPRESENTATION, WARRANTY, OR COVENANT CONTAINED IN THESE LEGAL TERMS OR OF ANY APPLICABLE LAW OR REGULATION. WE MAY TERMINATE YOUR USE OR PARTICIPATION IN THE SERVICES OR DELETE YOUR ACCOUNT AND ANY CONTENT OR INFORMATION THAT YOU POSTED AT ANY TIME, WITHOUT WARNING, IN OUR SOLE DISCRETION.
If we terminate or suspend your account for any reason, you are prohibited from registering and creating a new account under your name, a fake or borrowed name, or the name of any third party, even if you may be acting on behalf of the third party. In addition to terminating or suspending your account, we reserve the right to take appropriate legal action, including without limitation pursuing civil, criminal, and injunctive redress.
"},{"location":"terms/#16-modifications-and-interruptions","title":"16. Modifications and interruptions","text":"We reserve the right to change, modify, or remove the contents of the Services at any time or for any reason at our sole discretion without notice. However, we have no obligation to update any information on our Services. We will not be liable to you or any third party for any modification, price change, suspension, or discontinuance of the Services.
We cannot guarantee the Services will be available at all times. We may experience hardware, software, or other problems or need to perform maintenance related to the Services, resulting in interruptions, delays, or errors. We reserve the right to change, revise, update, suspend, discontinue, or otherwise modify the Services at any time or for any reason without notice to you. You agree that we have no liability whatsoever for any loss, damage, or inconvenience caused by your inability to access or use the Services during any downtime or discontinuance of the Services. Nothing in these Legal Terms will be construed to obligate us to maintain and support the Services or to supply any corrections, updates, or releases in connection therewith.
"},{"location":"terms/#17-governing-law","title":"17. Governing law","text":"These Legal Terms are governed by and interpreted following the laws of Germany, and the use of the United Nations Convention of Contracts for the International Sales of Goods is expressly excluded. If your habitual residence is in the EU, and you are a consumer, you additionally possess the protection provided to you by obligatory provisions of the law in your country to residence. dstack GmbH and yourself both agree to submit to the non-exclusive jurisdiction of the courts of Bayern, which means that you may make a claim to defend your consumer protection rights in regards to these Legal Terms in Germany, or in the EU country in which you reside.
"},{"location":"terms/#18-dispute-resolution","title":"18. Dispute resolution","text":"Informal Negotiations
To expedite resolution and control the cost of any dispute, controversy, or claim related to these Legal Terms (each a \" Dispute\" and collectively, the \"Disputes\") brought by either you or us (individually, a \"Party\" and collectively, the \" Parties\"), the Parties agree to first attempt to negotiate any Dispute (except those Disputes expressly provided below) informally for at least thirty (30) days before initiating arbitration. Such informal negotiations commence upon written notice from one Party to the other Party.
Binding Arbitration
Any dispute arising from the relationships between the Parties to these Legal Terms shall be determined by one arbitrator who will be chosen in accordance with the Arbitration and Internal Rules of the European Court of Arbitration being part of the European Centre of Arbitration having its seat in Strasbourg, and which are in force at the time the application for arbitration is filed, and of which adoption of this clause constitutes acceptance. The seat of arbitration shall be Munich , Germany . The language of the proceedings shall be German . Applicable rules of substantive law shall be the law of Germany .
Restrictions
The Parties agree that any arbitration shall be limited to the Dispute between the Parties individually. To the full extent permitted by law, (a) no arbitration shall be joined with any other proceeding; (b) there is no right or authority for any Dispute to be arbitrated on a class-action basis or to utilize class action procedures; and (c) there is no right or authority for any Dispute to be brought in a purported representative capacity on behalf of the general public or any other persons.
Exceptions to Informal Negotiations and Arbitration
The Parties agree that the following Disputes are not subject to the above provisions concerning informal negotiations binding arbitration: (a) any Disputes seeking to enforce or protect, or concerning the validity of, any of the intellectual property rights of a Party; (b) any Dispute related to, or arising from, allegations of theft, piracy, invasion of privacy, or unauthorized use; and (c) any claim for injunctive relief. If this provision is found to be illegal or unenforceable, then neither Party will elect to arbitrate any Dispute falling within that portion of this provision found to be illegal or unenforceable and such Dispute shall be decided by a court of competent jurisdiction within the courts listed for jurisdiction above, and the Parties agree to submit to the personal jurisdiction of that court.
"},{"location":"terms/#19-corrections","title":"19. Corrections","text":"There may be information on the Services that contains typographical errors, inaccuracies, or omissions, including descriptions, pricing, availability, and various other information. We reserve the right to correct any errors, inaccuracies, or omissions and to change or update the information on the Services at any time, without prior notice.
"},{"location":"terms/#20-disclaimer","title":"20. Disclaimer","text":"THE SERVICES ARE PROVIDED ON AN AS-IS AND AS-AVAILABLE BASIS. YOU AGREE THAT YOUR USE OF THE SERVICES WILL BE AT YOUR SOLE RISK. TO THE FULLEST EXTENT PERMITTED BY LAW, WE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, IN CONNECTION WITH THE SERVICES AND YOUR USE THEREOF, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT. WE MAKE NO WARRANTIES OR REPRESENTATIONS ABOUT THE ACCURACY OR COMPLETENESS OF THE SERVICES' CONTENT OR THE CONTENT OF ANY WEBSITES OR MOBILE APPLICATIONS LINKED TO THE SERVICES AND WE WILL ASSUME NO LIABILITY OR RESPONSIBILITY FOR ANY (1) ERRORS, MISTAKES, OR INACCURACIES OF CONTENT AND MATERIALS, (2) PERSONAL INJURY OR PROPERTY DAMAGE, OF ANY NATURE WHATSOEVER, RESULTING FROM YOUR ACCESS TO AND USE OF THE SERVICES, (3) ANY UNAUTHORIZED ACCESS TO OR USE OF OUR SECURE SERVERS AND/OR ANY AND ALL PERSONAL INFORMATION AND/OR FINANCIAL INFORMATION STORED THEREIN, (4) ANY INTERRUPTION OR CESSATION OF TRANSMISSION TO OR FROM THE SERVICES, (5) ANY BUGS, VIRUSES, TROJAN HORSES, OR THE LIKE WHICH MAY BE TRANSMITTED TO OR THROUGH THE SERVICES BY ANY THIRD PARTY, AND/OR (6) ANY ERRORS OR OMISSIONS IN ANY CONTENT AND MATERIALS OR FOR ANY LOSS OR DAMAGE OF ANY KIND INCURRED AS A RESULT OF THE USE OF ANY CONTENT POSTED, TRANSMITTED, OR OTHERWISE MADE AVAILABLE VIA THE SERVICES. WE DO NOT WARRANT, ENDORSE, GUARANTEE, OR ASSUME RESPONSIBILITY FOR ANY PRODUCT OR SERVICE ADVERTISED OR OFFERED BY A THIRD PARTY THROUGH THE SERVICES, ANY HYPERLINKED WEBSITE, OR ANY WEBSITE OR MOBILE APPLICATION FEATURED IN ANY BANNER OR OTHER ADVERTISING, AND WE WILL NOT BE A PARTY TO OR IN ANY WAY BE RESPONSIBLE FOR MONITORING ANY TRANSACTION BETWEEN YOU AND ANY THIRD-PARTY PROVIDERS OF PRODUCTS OR SERVICES. AS WITH THE PURCHASE OF A PRODUCT OR SERVICE THROUGH ANY MEDIUM OR IN ANY ENVIRONMENT, YOU SHOULD USE YOUR BEST JUDGMENT AND EXERCISE CAUTION WHERE APPROPRIATE.
"},{"location":"terms/#21-limitations-of-liability","title":"21. Limitations of liability","text":"IN NO EVENT WILL WE OR OUR DIRECTORS, EMPLOYEES, OR AGENTS BE LIABLE TO YOU OR ANY THIRD PARTY FOR ANY DIRECT, INDIRECT, CONSEQUENTIAL, EXEMPLARY, INCIDENTAL, SPECIAL, OR PUNITIVE DAMAGES, INCLUDING LOST PROFIT, LOST REVENUE, LOSS OF DATA, OR OTHER DAMAGES ARISING FROM YOUR USE OF THE SERVICES, EVEN IF WE HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. NOTWITHSTANDING ANYTHING TO THE CONTRARY CONTAINED HEREIN, OUR LIABILITY TO YOU FOR ANY CAUSE WHATSOEVER AND REGARDLESS OF THE FORM OF THE ACTION, WILL AT ALL TIMES BE LIMITED TO THE LESSER OF THE AMOUNT PAID, IF ANY, BY YOU TO US DURING THE zero (0) MONTH PERIOD PRIOR TO ANY CAUSE OF ACTION ARISING OR $0.00 USD. CERTAIN US STATE LAWS AND INTERNATIONAL LAWS DO NOT ALLOW LIMITATIONS ON IMPLIED WARRANTIES OR THE EXCLUSION OR LIMITATION OF CERTAIN DAMAGES. IF THESE LAWS APPLY TO YOU, SOME OR ALL OF THE ABOVE DISCLAIMERS OR LIMITATIONS MAY NOT APPLY TO YOU, AND YOU MAY HAVE ADDITIONAL RIGHTS.
"},{"location":"terms/#22-indemnification","title":"22. Indemnification","text":"You agree to defend, indemnify, and hold us harmless, including our subsidiaries, affiliates, and all of our respective officers, agents, partners, and employees, from and against any loss, damage, liability, claim, or demand, including reasonable attorneys\u2019 fees and expenses, made by any third party due to or arising out of: (1) use of the Services; (2) breach of these Legal Terms; (3) any breach of your representations and warranties set forth in these Legal Terms; (4) your violation of the rights of a third party, including but not limited to intellectual property rights; or (5) any overt harmful act toward any other user of the Services with whom you connected via the Services. Notwithstanding the foregoing, we reserve the right, at your expense, to assume the exclusive defense and control of any matter for which you are required to indemnify us, and you agree to cooperate, at your expense, with our defense of such claims. We will use reasonable efforts to notify you of any such claim, action, or proceeding which is subject to this indemnification upon becoming aware of it.
"},{"location":"terms/#23-user-data","title":"23. User data","text":"We will maintain certain data that you transmit to the Services for the purpose of managing the performance of the Services, as well as data relating to your use of the Services. Although we perform regular routine backups of data, you are solely responsible for all data that you transmit or that relates to any activity you have undertaken using the Services. You agree that we shall have no liability to you for any loss or corruption of any such data, and you hereby waive any right of action against us arising from any such loss or corruption of such data.
"},{"location":"terms/#24-electronic-communications-transactions-and-signatures","title":"24. Electronic communications, transactions, and signatures","text":"Visiting the Services, sending us emails, and completing online forms constitute electronic communications. You consent to receive electronic communications, and you agree that all agreements, notices, disclosures, and other communications we provide to you electronically, via email and on the Services, satisfy any legal requirement that such communication be in writing. YOU HEREBY AGREE TO THE USE OF ELECTRONIC SIGNATURES, CONTRACTS, ORDERS, AND OTHER RECORDS, AND TO ELECTRONIC DELIVERY OF NOTICES, POLICIES, AND RECORDS OF TRANSACTIONS INITIATED OR COMPLETED BY US OR VIA THE SERVICES. You hereby waive any rights or requirements under any statutes, regulations, rules, ordinances, or other laws in any jurisdiction which require an original signature or delivery or retention of non-electronic records, or to payments or the granting of credits by any means other than electronic means.
"},{"location":"terms/#25-california-users-and-residents","title":"25. California users and residents","text":"If any complaint with us is not satisfactorily resolved, you can contact the Complaint Assistance Unit of the Division of Consumer Services of the California Department of Consumer Affairs in writing at 1625 North Market Blvd., Suite N 112, Sacramento, California 95834 or by telephone at (800) 952-5210 or (916) 445-1254.
"},{"location":"terms/#26-miscellaneous","title":"26. Miscellaneous","text":"These Legal Terms and any policies or operating rules posted by us on the Services or in respect to the Services constitute the entire agreement and understanding between you and us. Our failure to exercise or enforce any right or provision of these Legal Terms shall not operate as a waiver of such right or provision. These Legal Terms operate to the fullest extent permissible by law. We may assign any or all of our rights and obligations to others at any time. We shall not be responsible or liable for any loss, damage, delay, or failure to act caused by any cause beyond our reasonable control. If any provision or part of a provision of these Legal Terms is determined to be unlawful, void, or unenforceable, that provision or part of the provision is deemed severable from these Legal Terms and does not affect the validity and enforceability of any remaining provisions. There is no joint venture, partnership, employment or agency relationship created between you and us as a result of these Legal Terms or use of the Services. You agree that these Legal Terms will not be construed against us by virtue of having drafted them. You hereby waive any and all defenses you may have based on the electronic form of these Legal Terms and the lack of signing by the parties hereto to execute these Legal Terms.
"},{"location":"terms/#27-contact-us","title":"27. Contact us","text":"In order to resolve a complaint regarding the Services or to receive further information regarding use of the Services, please contact us at hello@dstack.ai.
"},{"location":"blog/","title":"Blog","text":""},{"location":"blog/archive/say-goodbye-to-managed-notebooks/","title":"Say goodbye to managed notebooks","text":"Data science and ML tools have made significant advancements in recent years. This blog post aims to examine the advantages of cloud dev environments (CDE) for ML engineers and compare them with web-based managed notebooks.
"},{"location":"blog/archive/say-goodbye-to-managed-notebooks/#notebooks-are-here-to-stay","title":"Notebooks are here to stay","text":"Jupyter notebooks are instrumental for interactive work with data. They provide numerous advantages such as high interactivity, visualization support, remote accessibility, and effortless sharing.
Managed notebook platforms, like Google Colab and AWS SageMaker have become popular thanks to their easy integration with clouds. With pre-configured environments, managed notebooks remove the need to worry about infrastructure.
"},{"location":"blog/archive/say-goodbye-to-managed-notebooks/#reproducibility-challenge","title":"Reproducibility challenge","text":"As the code evolves, it needs to be converted into Python scripts and stored in Git for improved organization and version control. Notebooks alone cannot handle this task, which is why they must be a part of a developer environment that also supports Python scripts and Git.
The JupyterLab project attempts to address this by turning notebooks into an IDE by adding a file browser, terminal, and Git support.
"},{"location":"blog/archive/say-goodbye-to-managed-notebooks/#ides-get-equipped-for-ml","title":"IDEs get equipped for ML","text":"Recently, IDEs have improved in their ability to support machine learning. They have started to combine the benefits of traditional IDEs and managed notebooks.
IDEs have upgraded their remote capabilities, with better SSH support. Additionally, they now offer built-in support for editing notebooks.
Two popular IDEs, VS Code and PyCharm, have both integrated remote capabilities and seamless notebook editing features.
"},{"location":"blog/archive/say-goodbye-to-managed-notebooks/#the-rise-of-app-ecosystem","title":"The rise of app ecosystem","text":"Notebooks have been beneficial for their interactivity and sharing features. However, there are new alternatives like Streamlit and Gradio that allow developers to build data apps using Python code. These frameworks not only simplify app-building but also enhance reproducibility by integrating with Git.
Hugging Face Spaces, for example, is a popular tool today for sharing Streamlit and Gradio apps with others.
"},{"location":"blog/archive/say-goodbye-to-managed-notebooks/#say-hello-to-cloud-dev-environments","title":"Say hello to cloud dev environments!","text":"Remote development within IDEs is becoming increasingly popular, and as a result, cloud dev environments have emerged as a new concept. Various managed services, such as Codespaces and GitPod, offer scalable infrastructure while maintaining the familiar IDE experience.
One such open-source tool is dstack, which enables you to define your dev environment declaratively as code and run it on any cloud.
type: dev-environment\nbuild:\n - apt-get update\n - apt-get install -y ffmpeg\n - pip install -r requirements.txt\nide: vscode\nWith this tool, provisioning the required hardware, setting up the pre-built environment (no Docker is needed), and fetching your local code is automated.
$ dstack run .\n\n RUN CONFIGURATION USER PROJECT INSTANCE SPOT POLICY\n honest-jellyfish-1 .dstack.yml peter gcp a2-highgpu-1g on-demand\n\nStarting SSH tunnel...\n\nTo open in VS Code Desktop, use one of these link:\n vscode://vscode-remote/ssh-remote+honest-jellyfish-1/workflow\n\nTo exit, press Ctrl+C.\nYou can securely access the cloud development environment with the desktop IDE of your choice.
Learn more
Check out our guide for running dev environments in your cloud.
"},{"location":"blog/dstack-sky-own-cloud-accounts/","title":"dstack Sky now allows using your own cloud accounts","text":"dstack Sky enables you to access GPUs from the global marketplace at the most competitive rates. However, sometimes you may want to use your own cloud accounts. With today's release, both options are now supported.
"},{"location":"blog/dstack-sky-own-cloud-accounts/#configure-backends","title":"Configure backends","text":"To use your own cloud account, open the project settings and edit the corresponding backend.
You can configure your cloud accounts for any of the supported providers, including AWS, GCP, Azure, TensorDock, Lambda, CUDO, RunPod, and Vast.ai.
Additionally, you can disable certain backends if you do not plan to use them.
Typically, if you prefer using your own cloud accounts, it's recommended that you use the open-source version of dstack. However, if you prefer not to host it yourself, now you can use dstack Sky with your own cloud accounts as well.
Seeking the cheapest on-demand and spot cloud GPUs? dstack Sky has you covered!
Need help, have a question, or just want to stay updated?
Join Discord
"},{"location":"blog/dstack-sky/","title":"Introducing dstack Sky","text":"Today we're previewing dstack Sky, a service built on top of dstack that enables you to get GPUs at competitive rates from a wide pool of providers.
dstack's CLI and APIdstack is an open-source tool designed for managing AI infrastructure across various cloud platforms. It's lighter and more specifically geared towards AI tasks compared to Kubernetes.
Due to its support for multiple cloud providers, dstack is frequently used to access on-demand and spot GPUs across multiple clouds. From our users, we've learned that managing various cloud accounts, quotas, and billing can be cumbersome.
To streamline this process, we introduce dstack Sky, a managed service that enables users to access GPUs from multiple providers through dstack \u2013 without needing an account in each cloud provider.
Instead of running dstack server yourself, you point dstack config to a project set up with dstack Sky.
$ dstack config --url https://sky.dstack.ai \\\n --project my-awesome-project \\\n --token ca1ee60b-7b3f-8943-9a25-6974c50efa75\nNow, you can use dstack's CLI or API \u2013 just like you would with your own cloud accounts.
$ dstack run . -b tensordock -b vastai\n\n # BACKEND REGION RESOURCES SPOT PRICE \n 1 vastai canada 16xCPU/64GB/1xRTX4090/1TB no $0.35\n 2 vastai canada 16xCPU/64GB/1xRTX4090/400GB no $0.34\n 3 tensordock us 8xCPU/48GB/1xRTX4090/480GB no $0.74\n ...\n Shown 3 of 50 offers, $0.7424 max\n\nContinue? [y/n]:\nBackends
dstack Sky supports the same backends as the open-source version, except that you don't need to set them up. By default, it uses all supported backends.
You can use both on-demand and spot instances without needing to manage quotas, as they are automatically handled for you.
With dstack Sky you can use all of dstack's features, incl. dev environments, tasks, services, and fleets.
To use services, the open-source version requires setting up a gateway with your own domain. dstack Sky comes with a pre-configured gateway.
$ dstack gateway list\n BACKEND REGION NAME ADDRESS DOMAIN DEFAULT\n aws eu-west-1 dstack 3.252.79.143 my-awesome-project.sky.dstack.ai \u2713\nIf you run it with dstack Sky, the service's endpoint will be available at https://<run name>.<project name>.sky.dstack.ai.
Let's say we define a service:
type: service\n# Deploys Mixtral 8x7B with Ollama\n\n# Serve model using Ollama's Docker image\nimage: ollama/ollama\ncommands:\n - ollama serve &\n - sleep 3\n - ollama pull mixtral\n - fg\nport: 11434\n\n# Configure hardware requirements\nresources:\n gpu: 48GB..80GB\n\n# Enable OpenAI compatible endpoint\nmodel:\n type: chat\n name: mixtral\n format: openai\nIf it has a model mapping, the model will be accessible at https://gateway.<project name>.sky.dstack.ai via the OpenAI compatible interface.
from openai import OpenAI\n\n\nclient = OpenAI(\n base_url=\"https://gateway.<project name>.sky.dstack.ai\",\n api_key=\"<dstack token>\"\n)\n\ncompletion = client.chat.completions.create(\n model=\"mixtral\",\n messages=[\n {\"role\": \"user\", \"content\": \"Compose a poem that explains the concept of recursion in programming.\"}\n ]\n)\n\nprint(completion.choices[0].message)\nNow, you can choose \u2014 either use dstack via the open-source version or via dstack Sky, or even use them side by side.
Credits
Are you an active contributor to the AI community? Request free dstack Sky credits.
dstack Sky is live on Product Hunt. Support it by giving it your vote!
Join Discord
"},{"location":"changelog/","title":"Blog","text":""},{"location":"docs/","title":"What is dstack?","text":"dstack is an open-source container orchestration engine for AI. It accelerates the development, training, and deployment of AI models, and simplifies the management of clusters.
dstack is easy to use with any cloud or on-prem servers. Supported cloud providers include AWS, GCP, Azure, OCI, Lambda, TensorDock, Vast.ai, RunPod, and CUDO. For using dstack with on-prem servers, see fleets.
dstack supports NVIDIA GPU and Google Cloud TPU out of the box.
Before using dstack, install the server and configure backends for each cloud account (or Kubernetes cluster) that you intend to use.
dstack supports three types of run configurations:
Each type of run configuration allows you to specify commands for execution, required compute resources, retry policies, auto-scaling rules, authorization settings, and more.
Configuration can be defined as YAML files within your repo.
"},{"location":"docs/#2-run-configurations","title":"2. Run configurations","text":"Run any defined configuration either via dstack CLI or API.
dstack automatically handles provisioning, interruptions, port-forwarding, auto-scaling, network, volumes, run failures, out-of-capacity errors, and more.
Use fleets to provision and manage clusters and instances, both in the cloud and on-prem.
"},{"location":"docs/#where-do-i-start","title":"Where do I start?","text":"Before scheduling a task or deploying a model, you may want to run code interactively. Dev environments allow you to provision a remote machine set up with your code and favorite IDE with just one command.
"},{"location":"docs/dev-environments/#configuration","title":"Configuration","text":"First, create a YAML file in your project folder. Its name must end with .dstack.yml (e.g. .dstack.yml or dev.dstack.yml are both acceptable).
type: dev-environment\n\n# Specify the Python version, or your Docker image\npython: \"3.11\"\n\n# This pre-configures the IDE with required extensions\nide: vscode\n\n# Specify GPU, disk, and other resource requirements\nresources:\n gpu: 80GB\nIf you don't specify your Docker image, dstack uses the base image (pre-configured with Python, Conda, and essential CUDA drivers).
Reference
See the .dstack.yml reference for all supported configuration options and examples.
"},{"location":"docs/dev-environments/#running","title":"Running","text":"To run a configuration, use the dstack run command followed by the working directory path, configuration file path, and other options.
$ dstack run . -f .dstack.yml\n\n BACKEND REGION RESOURCES SPOT PRICE\n tensordock unitedkingdom 10xCPU, 80GB, 1xA100 (80GB) no $1.595\n azure westus3 24xCPU, 220GB, 1xA100 (80GB) no $3.673\n azure westus2 24xCPU, 220GB, 1xA100 (80GB) no $3.673\n\nContinue? [y/n]: y\n\nProvisioning `fast-moth-1`...\n---> 100%\n\nTo open in VS Code Desktop, use this link:\n vscode://vscode-remote/ssh-remote+fast-moth-1/workflow\nWhen dstack provisions the dev environment, it mounts the project folder contents.
If there are large files or folders you'd like to avoid uploading, you can list them in .gitignore.
By default, dstack run reuses idle instances from one of the existing fleets. If no idle instances meet the requirements, it creates a new fleet using one of the configured backends.
To have the fleet deleted after a certain idle time automatically, set termination_idle_time. By default, it's set to 5min.
Reference
See the CLI reference for more details on how dstack run works.
To open the dev environment in your desktop IDE, use the link from the output (such as vscode://vscode-remote/ssh-remote+fast-moth-1/workflow).
Alternatively, while the CLI is attached to the run, you can connect to the dev environment via SSH:
$ ssh fast-moth-1\nThe dstack ps command lists all running runs and their status.
Once the run exceeds the max duration, or when you use dstack stop, the dev environment and its cloud resources are deleted.
.dstack.yml reference for more details and examplesFleets enable efficient provisioning and management of clusters and instances, both in the cloud and on-prem. Once a fleet is created, it can be reused by dev environments, tasks, and services.
Fleets is a new feature. To use it, ensure you've installed version 0.18.6 or higher.
To create a fleet, create a YAML file in your project folder. Its name must end with .dstack.yml (e.g. .dstack.yml or fleet.dstack.yml are both acceptable).
To provision a fleet in the cloud using the configured backends, specify the required resources, number of nodes, and other optional parameters.
type: fleet\nname: my-fleet\nplacement: cluster\nnodes: 2\nresources:\n gpu: 24GB\nSet placement to cluster if the nodes should be interconnected (e.g. if you'd like to use them for multi-node tasks). In that case, dstack will provision all nodes in the same backend and region.
Defining fleets with YAML isn't supported yet for the kubernetes, vastai, and runpod backends.
To create a fleet from on-prem servers, specify their hosts along with the user, port, and SSH key for connection via SSH.
type: fleet\nname: my-fleet\nplacement: cluster\nssh_config:\n user: ubuntu\n identity_file: ~/.ssh/id_rsa\n hosts:\n - 3.255.177.51\n - 3.255.177.52\nRequirements
The on-prem servers should be pre-installed with CUDA 12.1 and NVIDIA Docker. The user should have sudo access.
Set placement to cluster if the nodes are interconnected (e.g. if you'd like to use them for multi-node tasks). In that case, by default, dstack will automatically detect the private network. You can specify the network parameter manually.
Reference
See the .dstack.yml reference for all supported configuration options and examples.
"},{"location":"docs/fleets/#creating-and-updating-fleets","title":"Creating and updating fleets","text":"To create or update the fleet, simply call the dstack apply command:
$ dstack apply -f my-gcp-fleet.dstack.yml\nFleet my-fleet does not exist yet. Create the fleet? [y/n]: y\n FLEET INSTANCE BACKEND RESOURCES PRICE STATUS CREATED \n my-fleet 0 pending now \n 1 pending now \nOnce the status of instances change to idle, they can be used by dstack run.
By default, dstack run tries to reuse idle instances from existing fleets. If no idle instances meet the requirements, dstack run creates a new fleet automatically. To avoid creating new fleet, specify pass --reuse to dstack run.
If you want a fleet to be automatically deleted after a certain idle time, you can set the you can set the termination_idle_time property.
The dstack fleet command lists fleet instances and theri status:
$ dstack fleet\n FLEET INSTANCE BACKEND GPU PRICE STATUS CREATED \n my-fleet 0 gcp (europe-west-1) L4:24GB (spot) $0.1624 idle 3 mins ago \n 1 gcp (europe-west-1) L4:24GB (spot) $0.1624 idle 3 mins ago \nWhen a fleet isn't used by run, you can delete it via dstack delete:
$ dstack delete -f cluster.dstack.yaml\nDelete the fleet my-gcp-fleet? [y/n]: y\nFleet my-gcp-fleet deleted\nYou can pass either the path to the configuration file or the fleet name directly.
To terminate and delete specific instances from a fleet, pass -i INSTANCE_NUM.
Before using dstack, install the server and configure backends.
To use dstack's CLI in a folder, first run dstack init within that folder.
$ mkdir quickstart && cd quickstart\n$ dstack init\nYour folder can be a regular local folder or a Git repo.
"},{"location":"docs/quickstart/#define-a-configuration","title":"Define a configuration","text":"Define what you want to run as a YAML file. The filename must end with .dstack.yml (e.g., .dstack.yml or train.dstack.yml are both acceptable).
Dev environments allow you to quickly provision a machine with a pre-configured environment, resources, IDE, code, etc.
type: dev-environment\n\n# Use either `python` or `image` to configure environment\npython: \"3.11\"\n# image: ghcr.io/huggingface/text-generation-inference:latest\n\nide: vscode\n\n# (Optional) Configure `gpu`, `memory`, `disk`, etc\nresources:\n gpu: 24GB\nTasks make it very easy to run any scripts, be it for training, data processing, or web apps. They allow you to pre-configure the environment, resources, code, etc.
type: task\n\npython: \"3.11\"\nenv:\n - HF_HUB_ENABLE_HF_TRANSFER=1\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py\n\n# (Optional) Configure `gpu`, `memory`, `disk`, etc\nresources:\n gpu: 24GB\nEnsure requirements.txt and train.py are in your folder. You can take them from examples.
Services make it easy to deploy models and apps cost-effectively as public endpoints, allowing you to use any frameworks.
type: service\n\nimage: ghcr.io/huggingface/text-generation-inference:latest\nenv:\n - HUGGING_FACE_HUB_TOKEN # required to run gated models\n - MODEL_ID=mistralai/Mistral-7B-Instruct-v0.1\ncommands:\n - text-generation-launcher --port 8000 --trust-remote-code\nport: 8000\n\n# (Optional) Configure `gpu`, `memory`, `disk`, etc\nresources:\n gpu: 24GB\nRun a configuration using the dstack run command, followed by the working directory path (e.g., .), and the path to the configuration file.
$ dstack run . -f train.dstack.yml\n\n BACKEND REGION RESOURCES SPOT PRICE\n tensordock unitedkingdom 10xCPU, 80GB, 1xA100 (80GB) no $1.595\n azure westus3 24xCPU, 220GB, 1xA100 (80GB) no $3.673\n azure westus2 24xCPU, 220GB, 1xA100 (80GB) no $3.673\n\nContinue? [y/n]: y\n\nProvisioning...\n---> 100%\n\nEpoch 0: 100% 1719/1719 [00:18<00:00, 92.32it/s, loss=0.0981, acc=0.969]\nEpoch 1: 100% 1719/1719 [00:18<00:00, 92.32it/s, loss=0.0981, acc=0.969]\nEpoch 2: 100% 1719/1719 [00:18<00:00, 92.32it/s, loss=0.0981, acc=0.969]\nThe dstack run command automatically uploads your code, including any local uncommitted changes.
Fleets
By default, dstack run reuses idle instances from one of the existing fleets. If no idle instances meet the requirements, it creates a new fleet using one of the configured backends.
Services make it easy to deploy models and web applications as public, secure, and scalable endpoints. They are provisioned behind a gateway that automatically provides an HTTPS domain, handles authentication, distributes load, and performs auto-scaling.
GatewaysIf you're using the open-source server, you must set up a gateway before you can run a service.
If you're using dstack Sky , the gateway is already set up for you.
"},{"location":"docs/services/#configuration","title":"Configuration","text":"First, create a YAML file in your project folder. Its name must end with .dstack.yml (e.g. .dstack.yml or serve.dstack.yml are both acceptable).
type: service\n\npython: \"3.11\"\nenv:\n - MODEL=NousResearch/Llama-2-7b-chat-hf\ncommands:\n - pip install vllm\n - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000\nport: 8000\n\nresources:\n gpu: 80GB\n\n# (Optional) Enable the OpenAI-compatible endpoint\nmodel:\n format: openai\n type: chat\n name: NousResearch/Llama-2-7b-chat-hf\nIf you don't specify your Docker image, dstack uses the base image (pre-configured with Python, Conda, and essential CUDA drivers).
Auto-scaling
By default, the service is deployed to a single instance. However, you can specify the number of replicas and scaling policy. In this case, dstack auto-scales it based on the load.
Reference
See the .dstack.yml reference for all supported configuration options and examples.
"},{"location":"docs/services/#running","title":"Running","text":"To run a configuration, use the dstack run command followed by the working directory path, configuration file path, and any other options.
$ dstack run . -f serve.dstack.yml\n\n BACKEND REGION RESOURCES SPOT PRICE\n tensordock unitedkingdom 10xCPU, 80GB, 1xA100 (80GB) no $1.595\n azure westus3 24xCPU, 220GB, 1xA100 (80GB) no $3.673\n azure westus2 24xCPU, 220GB, 1xA100 (80GB) no $3.673\n\nContinue? [y/n]: y\n\nProvisioning...\n---> 100%\n\nService is published at https://yellow-cat-1.example.com\nWhen deploying the service, dstack run mounts the current folder's contents.
If there are large files or folders you'd like to avoid uploading, you can list them in .gitignore.
By default, dstack run reuses idle instances from one of the existing fleets. If no idle instances meet the requirements, it creates a new fleet using one of the configured backends.
To have the fleet deleted after a certain idle time automatically, set termination_idle_time. By default, it's set to 5min.
Reference
See the CLI reference for more details on how dstack run works.
One the service is up, its endpoint is accessible at https://<run name>.<gateway domain>.
By default, the service endpoint requires the Authorization header with Bearer <dstack token>.
$ curl https://yellow-cat-1.example.com/v1/chat/completions \\\n -H 'Content-Type: application/json' \\\n -H 'Authorization: Bearer <dstack token>' \\\n -d '{\n \"model\": \"NousResearch/Llama-2-7b-chat-hf\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Compose a poem that explains the concept of recursion in programming.\"\n }\n ]\n }'\nAuthorization can be disabled by setting auth to false in the service configuration file.
In case the service has the model mapping configured, you will also be able to access the model at https://gateway.<gateway domain> via the OpenAI-compatible interface.
The dstack ps command lists all running runs and their status.
When you use dstack stop, the service and its cloud resources are deleted.
.dstack.yml reference for more details and examplesTasks allow for convenient scheduling of various batch jobs, such as training, fine-tuning, or data processing. They can also be used to run web applications when features offered by services are not needed, such as for debugging.
You can run tasks on a single machine or on a cluster of nodes.
"},{"location":"docs/tasks/#configuration","title":"Configuration","text":"First, create a YAML file in your project folder. Its name must end with .dstack.yml (e.g. .dstack.yml or train.dstack.yml are both acceptable).
type: task\n\npython: \"3.11\"\nenv:\n - HF_HUB_ENABLE_HF_TRANSFER=1\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - tensorboard --logdir results/runs &\n - python fine-tuning/qlora/train.py\nports:\n - 6000\n\n# (Optional) Configure `gpu`, `memory`, `disk`, etc\nresources:\n gpu: 80GB\nIf you don't specify your Docker image, dstack uses the base image (pre-configured with Python, Conda, and essential CUDA drivers).
Distributed tasks
By default, tasks run on a single instance. However, you can specify the number of nodes. In this case, dstack provisions a cluster of instances.
Reference
See the .dstack.yml reference for all supported configuration options and examples.
"},{"location":"docs/tasks/#running","title":"Running","text":"To run a configuration, use the dstack run command followed by the working directory path, configuration file path, and other options.
$ dstack run . -f train.dstack.yml\n\n BACKEND REGION RESOURCES SPOT PRICE\n tensordock unitedkingdom 10xCPU, 80GB, 1xA100 (80GB) no $1.595\n azure westus3 24xCPU, 220GB, 1xA100 (80GB) no $3.673\n azure westus2 24xCPU, 220GB, 1xA100 (80GB) no $3.673\n\nContinue? [y/n]: y\n\nProvisioning...\n---> 100%\n\nTensorBoard 2.13.0 at http://localhost:6006/ (Press CTRL+C to quit)\n\nEpoch 0: 100% 1719/1719 [00:18<00:00, 92.32it/s, loss=0.0981, acc=0.969]\nEpoch 1: 100% 1719/1719 [00:18<00:00, 92.32it/s, loss=0.0981, acc=0.969]\nEpoch 2: 100% 1719/1719 [00:18<00:00, 92.32it/s, loss=0.0981, acc=0.969]\nIf the task specifies ports, dstack run automatically forwards them to your local machine for convenient and secure access.
When running the task, dstack run mounts the current folder's contents.
If there are large files or folders you'd like to avoid uploading, you can list them in .gitignore.
By default, dstack run reuses idle instances from one of the existing fleets. If no idle instances meet the requirements, it creates a new fleet using one of the configured backends.
To have the fleet deleted after a certain idle time automatically, set termination_idle_time. By default, it's set to 5min.
Reference
See the CLI reference for more details on how dstack run works.
The dstack ps command lists all running runs and their status.
Once you use dstack stop (or when the run exceeds the max_duration), the instances return to the fleet.
.dstack.yml reference for more details and examplesGateways handle the ingress traffic of running services. They provide services with HTTPS domains, handle authentication, distribute load, and perform auto-scaling. In order to run a service, you need to have at least one gateway set up.
If you're using dstack Sky , the gateway is already set up for you.
"},{"location":"docs/concepts/gateways/#configuration","title":"Configuration","text":"First, create a YAML file in your project folder. Its name must end with .dstack.yml (e.g. .dstack.yml or gateway.dstack.yml are both acceptable).
type: gateway\nname: example-gateway\n\nbackend: aws\nregion: eu-west-1\ndomain: example.com\nA domain name is required to create a gateway.
Reference
See the .dstack.yml reference for all supported configuration options and examples.
"},{"location":"docs/concepts/gateways/#creating-and-updating-gateways","title":"Creating and updating gateways","text":"To create or update the gateway, simply call the dstack apply command:
$ dstack apply . -f examples/deployment/gateway.dstack.yml\n\nThe example-gateway doesn't exist. Create it? [y/n]: y\n\n BACKEND REGION NAME HOSTNAME DOMAIN DEFAULT STATUS\n aws eu-west-1 example-gateway example.com \u2713 submitted\nOnce the gateway is assigned a hostname, go to your domain's DNS settings and add an A DNS record for *.<gateway domain> (e.g., *.example.com) pointing to the gateway's hostname.
This will allow you to access runs and models using this domain.
"},{"location":"docs/concepts/gateways/#managing-gateways","title":"Managing gateways","text":""},{"location":"docs/concepts/gateways/#listing-gateways","title":"Listing gateways","text":"The dstack gateway list command lists existing gateways and their status.
To delete a gateway, pass gateway configuration to dstack delete:
$ dstack delete . -f examples/deployment/gateway.dstack.yml\n.dstack.yml reference for more details and examplesPools enable the efficient reuse of cloud instances and on-premises servers across runs, simplifying their management.
"},{"location":"docs/concepts/pools/#adding-instances","title":"Adding instances","text":""},{"location":"docs/concepts/pools/#automatic-provisioning","title":"Automatic provisioning","text":"By default, when using the dstack run command, it tries to reuse an instance from a pool. If no idle instance meets the requirements, dstack automatically provisions a new cloud instance and adds it to the pool.
To avoid provisioning new cloud instances with dstack run, use --reuse. Your run will be assigned to an idle instance in the pool. If there are no available idle instances in the pool, the run will fail.
By default, dstack run sets the idle duration of a newly provisioned instance to 5m. This means that if the run is finished and the instance remains idle for longer than five minutes, it is automatically removed from the pool. To override the default idle duration, use --idle-duration DURATION with dstack run.
To manually provision a cloud instance and add it to a pool, use dstack pool add:
$ dstack pool add --gpu 80GB\n\n BACKEND REGION RESOURCES SPOT PRICE\n tensordock unitedkingdom 10xCPU, 80GB, 1xA100 (80GB) no $1.595\n azure westus3 24xCPU, 220GB, 1xA100 (80GB) no $3.673\n azure westus2 24xCPU, 220GB, 1xA100 (80GB) no $3.673\n\nContinue? [y/n]: y\nThe dstack pool add command allows specifying resource requirements, along with the spot policy, idle duration, max price, retry policy, and other policies.
The default idle duration if you're using dstack pool add is 72h. To override it, use the --idle-duration DURATION argument.
You can also specify the policies via .dstack/profiles.yml instead of passing them as arguments. For more details on policies and their defaults, refer to .dstack/profiles.yml.
The dstack pool add command is not supported for Kubernetes, VastAI, and RunPod backends yet.
Any on-prem server that can be accessed via SSH can be added to a pool and used to run workloads.
To add on-prem servers to the pool, use the dstack pool add-ssh command and pass the hostname of your server along with the SSH key.
$ dstack pool add-ssh -i ~/.ssh/id_rsa ubuntu@54.73.155.119\nThe command accepts the same arguments as the standard ssh command.
Requirements
The on-prem server should be pre-installed with CUDA 12.1 and NVIDIA Docker.
Once the instance is provisioned, you'll see it in the pool and will be able to run workloads on it.
"},{"location":"docs/concepts/pools/#clusters","title":"Clusters","text":"If you want on-prem instances to run multi-node tasks, ensure these on-prem servers share the same private network. Additionally, you need to pass the --network option to dstack pool add-ssh:
$ dstack pool add-ssh -i ~/.ssh/id_rsa ubuntu@54.73.155.119 \\\n --network 10.0.0.0/24\nThe --network argument accepts the IP address range (CIDR) of the private network of the instance.
Once you've added multiple instances with the same network value, you can use them as a cluster to run multi-node tasks.
"},{"location":"docs/concepts/pools/#removing-instances","title":"Removing instances","text":"If the instance remains idle for the configured idle duration, dstack removes it and deletes all cloud resources.
To remove an instance from the pool manually, use the dstack pool rm command.
$ dstack pool rm <instance name>\nThe dstack pool ps command lists active instances and their status (busy or idle).
Volumes allow you to persist data between runs. dstack simplifies managing volumes and lets you mount them to a specific directory when working with dev environments, tasks, and services.
Experimental
Volumes are currently experimental and only work with the aws and runpod backends. Support for other backends is coming soon.
First, create a YAML file in your project folder. Its name must end with .dstack.yml (e.g. .dstack.yml or vol.dstack.yml are both acceptable).
type: volume\nname: my-new-volume\nbackend: aws\nregion: eu-central-1\nsize: 100GB\nIf you use this configuration, dstack will create a new volume based on the specified options.
Registering existing volumes
If you prefer not to create a new volume but to reuse an existing one (e.g., created manually), you can specify its ID via volume_id. In this case, dstack will register the specified volume so that you can use it with dev environments, tasks, and services.
Reference
See the .dstack.yml reference for all supported configuration options and examples.
"},{"location":"docs/concepts/volumes/#creating-and-registering-volumes","title":"Creating and registering volumes","text":"To create or register the volume, simply call the dstack apply command:
$ dstack apply -f volume.dstack.yml\nVolume my-new-volume does not exist yet. Create the volume? [y/n]: y\n NAME BACKEND REGION STATUS CREATED \n my-new-volume aws eu-central-1 submitted now \nWhen creating the volume dstack automatically creates an ext4 file system on it.
Once created, the volume can be attached with dev environments, tasks, and services.
"},{"location":"docs/concepts/volumes/#attaching-volumes","title":"Attaching volumes","text":"Dev environments, tasks, and services let you attach any number of volumes. To attach a volume, simply specify its name using the volumes property and specify where to mount its contents:
type: dev-environment\nide: vscode\nvolumes:\n - name: my-new-volume\n path: /volume_data\nOnce you run this configuration, the contents of the volume will be attached to /volume_data inside the dev environment, and its contents will persist across runs.
Limitations
When you're running a dev environment, task, or service with dstack, it automatically mounts the project folder contents to /workflow (and sets that as the current working directory). Right now, dstack doesn't allow you to attach volumes to /workflow or any of its subdirectories.
The dstack volume list command lists created and registered volumes:
$ dstack volume list\nNAME BACKEND REGION STATUS CREATED\n my-new-volume aws eu-central-1 active 3 weeks ago\nWhen the volume isn't attached to any active dev environment, task, or service, you can delete it using dstack delete:
$ dstack delete -f vol.dstack.yaml\nIf the volume was created using dstack, it will be physically destroyed along with the data. If you've registered an existing volume, it will be de-registered with dstack but will keep the data.
Since volumes are backed up by cloud network disks, you can only use them within the same cloud. If you need to access data across different backends, you should either use object storage or replicate the data across multiple volumes.
Using volumes across regionsTypically, network volumes are associated with specific regions, so you can't use them in other regions. Often, volumes are also linked to availability zones, but some providers support volumes that can be used across different availability zones within the same region.
Attaching volumes to multiple runs and instancesYou can mount a volume in multiple runs. This feature is currently supported only by the runpod backend.
Below are tips and tricks to use dstack more efficiently.
Before running a task or service, it's recommended that you first start with a dev environment. Dev environments allow you to run commands interactively.
Once the commands work, go ahead and run them as a task or a service.
NotebooksVS Code
When you access a dev environment using your desktop VS Code, it allows you to work with Jupyter notebooks via its pre-configured and easy-to-use extension.
JupyterLab
If you prefer to use JupyterLab, you can run it as a task:
type: task\n\ncommands:\n - pip install jupyterlab\n - jupyter lab --allow-root\n\nports:\n - 8888\nTasks can be used not only for batch jobs but also for web applications.
type: task\n\npython: \"3.11\"\n\ncommands:\n - pip3 install streamlit\n - streamlit hello\n\nports: \n - 8501\nWhile you run a task, dstack forwards the remote ports to localhost.
$ dstack run . -f app.dstack.yml\n\n Welcome to Streamlit. Check out our demo in your browser.\n\n Local URL: http://localhost:8501\nThis allows you to access the remote 8501 port on localhost:8501 while the CLI is attached.
If you want to override the local port, use the --port option:
$ dstack run . -f app.dstack.yml --port 3000:8501\nThis will forward the remote 8501 port to localhost:3000.
Services require a gateway but they also provide additional features for production-grade service deployment not offered by tasks, such as HTTPS domains and auto-scaling. If you run a web app as a task and it works, go ahead and run it as a service.
"},{"location":"docs/guides/protips/#environment-variables","title":"Environment variables","text":"If a configuration requires an environment variable that you don't want to hardcode in the YAML, you can define it without assigning a value:
type: dev-environment\n\nenv:\n - HUGGING_FACE_HUB_TOKEN\n\npython: \"3.11\"\nide: vscode\nThen, you can pass the environment variable either via the shell:
HUGGING_FACE_HUB_TOKEN=... dstack run . -f .dstack.yml\nOr via the -e option of the dstack run command:
dstack run . -f .dstack.yml -e HUGGING_FACE_HUB_TOKEN=...\nA better way to configure environment variables not hardcoded in YAML is by specifying them in a .env file:
HUGGING_FACE_HUB_TOKEN=...\nIf you install direnv , it will automatically pass the environment variables from the .env file to the dstack run command.
Remember to add .env to .gitignore to avoid pushing it to the repo.
dstack has support for volumes to persist data across different runs and instance interruptions. Volumes are ideal for storing intermediate work and data that should be quickly accessible.
You can also load and save data using an object storage like S3 or HuggingFace Datasets. For models, it's best to use services like HuggingFace Hub. dstack has no explicit support for object storage. You can load and save data directly from your code.
By default, the dstack run command reuses an idle instance from the pool. If no instance matches the requirements, it creates a new one.
When the run finishes, the instance remains idle for the configured time (by default, 5m) before it gets destroyed.
You can change the default idle duration by using --idle-duration DURATION with dstack run, or set termination_idle_duration in the configuration or profile.
An idle instance can be destroyed at any time via dstack pool rm INSTANCE_NAME.
If you don't want to specify the same parameters for each configuration, you can define them once via profiles and reuse them across configurations.
This can be handy, for example, for configuring parameters such as max_duration, max_price, termination_idle_duration, regions, etc.
Set default to true in your profile, and it will be applied automatically to any run.
By default, dstack run runs in attached mode. This means it streams the logs as they come in and, in the case of a task, forwards its ports to localhost.
If you detach the CLI, you can re-attach it using dstack logs -a RUN_NAME.
To run in detached mode, use -d with dstack run.
dstack natively supports NVIDIA GPU, and Google Cloud TPU accelerator chips.
The gpu property withing resources (or the --gpu option with dstack run) allows specifying not only memory size but also GPU names, their memory, and quantity.
Examples:
1 (any GPU)A100 (A100)24GB.. (any GPU starting from 24GB)24GB..40GB:2 (two GPUs between 24GB and 40GB)A10G,A100 (either A10G or A100)A100:80GB (one A100 of 80GB)A100:2 (two A100)A100:40GB:2 (two A100 40GB)tpu-v2-8 (v2 with 8 TPU cores)Currently, you can't specify other than 8 TPU cores. This means only single host workloads are supported. Support for multiple hosts is coming soon.
"},{"location":"docs/guides/protips/#service-quotas","title":"Service quotas","text":"If you're using your own AWS, GCP, Azure, or OCI accounts, before you can use GPUs or spot instances, you have to request the corresponding service quotas for each type of instance in each region.
AWSCheck this guide on EC2 service quotas. The relevant service quotas include:
Running On-Demand P instances (on-demand V100, A100 80GB x8)All P4, P3 and P2 Spot Instance Requests (spot V100, A100 80GB x8)Running On-Demand G and VT instances (on-demand T4, A10G, L4)All G and VT Spot Instance Requests (spot T4, A10G, L4)Running Dedicated p5 Hosts (on-demand H100)All P5 Spot Instance Requests (spot H100)Check this guide on Compute Engine service quotas. The relevant service quotas include:
NVIDIA V100 GPUs (on-demand V100)Preemtible V100 GPUs (spot V100)NVIDIA T4 GPUs (on-demand T4)Preemtible T4 GPUs (spot T4)NVIDIA L4 GPUs (on-demand L4)Preemtible L4 GPUs (spot L4)NVIDIA A100 GPUs (on-demand A100)Preemtible A100 GPUs (spot A100)NVIDIA A100 80GB GPUs (on-demand A100 80GB)Preemtible A100 80GB GPUs (spot A100 80GB)NVIDIA H100 GPUs (on-demand H100)Preemtible H100 GPUs (spot H100)Check this guide on Azure service quotas. The relevant service quotas include:
Total Regional Spot vCPUs (any spot instances)Standard NCASv3_T4 Family vCPUs (on-demand T4)Standard NVADSA10v5 Family vCPUs (on-demand A10)Standard NCADS_A100_v4 Family vCPUs (on-demand A100 80GB)Standard NDASv4_A100 Family vCPUs (on-demand A100 40GB x8)Standard NDAMSv4_A100Family vCPUs (on-demand A100 80GB x8)Standard NCadsH100v5 Family vCPUs (on-demand H100)Standard NDSH100v5 Family vCPUs (on-demand H100 x8)Check this guide on requesting OCI service limits increase. The relevant service category is compute. The relevant resources include:
GPUs for GPU.A10 based VM and BM instances (on-demand A10)GPUs for GPU2 based VM and BM instances (on-demand P100)GPUs for GPU3 based VM and BM instances (on-demand V100)Note, for AWS, GCP, and Azure, service quota values are measured with the number of CPUs rather than GPUs.
"},{"location":"docs/installation/","title":"Installation","text":"To use the open-source version of dstack with your own cloud accounts or on-prem servers, you have to set up the server.
Follow the steps below to set up the server.
"},{"location":"docs/installation/#1-configure-backends","title":"1. Configure backends","text":"If you want the dstack server to run containers or manage clusters in your cloud accounts (or use Kubernetes), create the ~/.dstack/server/config.yml file and configure backends.
Once the ~/.dstack/server/config.yml file is configured, proceed to start the server:
$ pip install \"dstack[all]\" -U\n$ dstack server\n\nApplying ~/.dstack/server/config.yml...\n\nThe admin token is \"bbae0f28-d3dd-4820-bf61-8f4bb40815da\"\nThe server is running at http://127.0.0.1:3000/\n$ docker run -p 3000:3000 \\\n -v $HOME/.dstack/server/:/root/.dstack/server \\\n dstackai/dstack\n\nApplying ~/.dstack/server/config.yml...\n\nThe admin token is \"bbae0f28-d3dd-4820-bf61-8f4bb40815da\"\nThe server is running at http://127.0.0.1:3000/\nFor more details on how to deploy dstack using Docker, check its Docker repo.
By default, the dstack server stores its state in ~/.dstack/server/data using SQLite. To use a database, set the DSTACK_DATABASE_URL environment variable.
The server can be set up anywhere: on your laptop, a dedicated server, or in the cloud. Once the dstack server is up, you can use the CLI or API.
To point the CLI to the dstack server, configure it with the server address, user token and project name:
$ pip install dstack\n$ dstack config --url http://127.0.0.1:3000 \\\n --project main \\\n --token bbae0f28-d3dd-4820-bf61-8f4bb40815da\n\nConfiguration is updated at ~/.dstack/config.yml\nThis configuration is stored in ~/.dstack/config.yml.
Fleets
If you want the dstack server to run containers on your on-prem servers, use fleets.
If you don't want to host the dstack server yourself or would like to access GPU from the dstack marketplace, sign up with dstack Sky .
If you've signed up, open your project settings, and copy the dstack config command to point the CLI to the project.
Then, install the CLI on your machine and use the copied command.
$ pip install dstack\n$ dstack config --url https://sky.dstack.ai \\\n --project peterschmidt85 \\\n --token bbae0f28-d3dd-4820-bf61-8f4bb40815da\n\nConfiguration is updated at ~/.dstack/config.yml\nBy default, dstack Sky uses the GPU from its marketplace, which requires a credit card to be attached in your account settings.
To use your own cloud accounts, click the settings icon of the corresponding backend and specify credentials:
"},{"location":"docs/installation/#whats-next","title":"What's next?","text":"dev-environmenttaskserviceSometimes, you may want to reuse the same parameters across different .dstack.yml configurations.
This can be achieved by defining those parameters in a profile.
Profiles can be defined on the repository level (via the .dstack/profiles.yml file in the root directory of the repository) or on the global level (via the ~/.dstack/profiles.yml file).
Any profile can be marked as default so that it will be applied automatically for any run. Otherwise, you can refer to a specific profile via --profile NAME in dstack run.
profiles:\n - name: my-profile\n\n # The spot pololicy can be \"spot\", \"on-demand\", or \"auto\"\n spot_policy: auto\n\n # Limit the maximum price of the instance per hour\n max_price: 1.5\n\n # Stop any run if it runs longer that this duration\n max_duration: 1d\n\n # Use only these backends\n backends: [azure, lambda]\n\n # If set to true, this profile will be applied automatically\n default: true\nThe profile configuration supports many properties. See below.
"},{"location":"docs/reference/profiles.yml/#root-reference","title":"Root reference","text":""},{"location":"docs/reference/profiles.yml/#backends","title":"backends - (Optional) The backends to consider for provisioning (e.g., [aws, gcp]).","text":""},{"location":"docs/reference/profiles.yml/#regions","title":"regions - (Optional) The regions to consider for provisioning (e.g., [eu-west-1, us-west4, westeurope]).","text":""},{"location":"docs/reference/profiles.yml/#instance_types","title":"instance_types - (Optional) The cloud-specific instance types to consider for provisioning (e.g., [p3.8xlarge, n1-standard-4]).","text":""},{"location":"docs/reference/profiles.yml/#spot_policy","title":"spot_policy - (Optional) The policy for provisioning spot or on-demand instances: spot, on-demand, or auto.","text":""},{"location":"docs/reference/profiles.yml/#_retry","title":"retry - (Optional) The policy for resubmitting the run. Defaults to false.","text":""},{"location":"docs/reference/profiles.yml/#_retry_policy","title":"retry_policy - (Optional) The policy for resubmitting the run. Deprecated in favor of retry.","text":""},{"location":"docs/reference/profiles.yml/#max_duration","title":"max_duration - (Optional) The maximum duration of a run (e.g., 2h, 1d, etc). After it elapses, the run is forced to stop. Defaults to off.","text":""},{"location":"docs/reference/profiles.yml/#max_price","title":"max_price - (Optional) The maximum instance price per hour, in dollars.","text":""},{"location":"docs/reference/profiles.yml/#pool_name","title":"pool_name - (Optional) The name of the pool. If not set, dstack will use the default name.","text":""},{"location":"docs/reference/profiles.yml/#instance_name","title":"instance_name - (Optional) The name of the instance.","text":""},{"location":"docs/reference/profiles.yml/#creation_policy","title":"creation_policy - (Optional) The policy for using instances from the pool. Defaults to reuse-or-create.","text":""},{"location":"docs/reference/profiles.yml/#termination_policy","title":"termination_policy - (Optional) The policy for instance termination. Defaults to destroy-after-idle.","text":""},{"location":"docs/reference/profiles.yml/#termination_idle_time","title":"termination_idle_time - (Optional) Time to wait before destroying the idle instance. Defaults to 5m for dstack run and to 3d for dstack pool add.","text":""},{"location":"docs/reference/profiles.yml/#name","title":"name - The name of the profile that can be passed as --profile to dstack run.","text":""},{"location":"docs/reference/profiles.yml/#default","title":"default - (Optional) If set to true, dstack run will use this profile by default..","text":""},{"location":"docs/reference/profiles.yml/#retry","title":"retry","text":""},{"location":"docs/reference/profiles.yml/#on_events","title":"on_events - The list of events that should be handled with retry. Supported events are no-capacity, interruption, and error.","text":""},{"location":"docs/reference/profiles.yml/#duration","title":"duration - (Optional) The maximum period of retrying the run, e.g., 4h or 1d.","text":""},{"location":"docs/reference/api/python/","title":"Python API","text":"The Python API enables running tasks, services, and managing runs programmatically.
"},{"location":"docs/reference/api/python/#usage-example","title":"Usage example","text":"Below is a quick example of submitting a task for running and displaying its logs.
import sys\n\nfrom dstack.api import Task, GPU, Client, Resources\n\nclient = Client.from_config()\n\ntask = Task(\n image=\"ghcr.io/huggingface/text-generation-inference:latest\",\n env={\"MODEL_ID\": \"TheBloke/Llama-2-13B-chat-GPTQ\"},\n commands=[\n \"text-generation-launcher --trust-remote-code --quantize gptq\",\n ],\n ports=[\"80\"],\n resources=Resources(gpu=GPU(memory=\"24GB\")),\n)\n\nrun = client.runs.submit(\n run_name=\"my-awesome-run\", # If not specified, a random name is assigned \n configuration=task,\n repo=None, # Specify to mount additional files\n)\n\nrun.attach()\n\ntry:\n for log in run.logs():\n sys.stdout.buffer.write(log)\n sys.stdout.buffer.flush()\nexcept KeyboardInterrupt:\n run.stop(abort=True)\nfinally:\n run.detach()\nNOTE:
configuration argument in the submit method can be either dstack.api.Task or dstack.api.Service. dstack.api.Task or dstack.api.Service, you may specify the image argument. If image isn't specified, the default image will be used. For a private Docker registry, ensure you also pass the registry_auth argument.repo argument in the submit method allows the mounting of a local folder, a remote repo, or a programmatically created repo. In this case, the commands argument can refer to the files within this repo.attach method waits for the run to start and, for dstack.api.Task sets up an SSH tunnel and forwards configured ports to localhost.dstack.api","text":""},{"location":"docs/reference/api/python/#dstack.api.Client","title":"dstack.api.Client","text":"High-level API client for interacting with dstack server
Attributes:
Name Type Descriptionruns RunCollection Operations with runs.
repos RepoCollection Operations with repositories.
backends BackendCollection Operations with backends.
"},{"location":"docs/reference/api/python/#dstack.api.Client.from_config","title":"from_config(project_name=None, server_url=None, user_token=None, ssh_identity_file=None) staticmethod","text":"Creates a Client using the default configuration from ~/.dstack/config.yml if it exists.
Parameters:
Name Type Description Defaultproject_name Optional[str] The name of the project, required if server_url and user_token are specified
None server_url Optional[str] The dstack server URL (e.g. http://localhost:3000/ or https://sky.dstack.ai)
None user_token Optional[str] The dstack user token
None ssh_identity_file Optional[PathLike] The private SSH key path for SSH tunneling
None Returns:
Type DescriptionClient A client instance
"},{"location":"docs/reference/api/python/#dstack.api.Client.runs","title":"dstack.api.RunCollection","text":"Operations with runs
"},{"location":"docs/reference/api/python/#dstack.api.RunCollection.get","title":"get(run_name)","text":"Get run by run name
Parameters:
Name Type Description Defaultrun_name str run name
requiredReturns:
Type DescriptionOptional[Run] The run or None if not found
list(all=False)","text":"List runs
Parameters:
Name Type Description Defaultall bool show all runs (active and finished) if True
False Returns:
Type DescriptionList[Run] list of runs
"},{"location":"docs/reference/api/python/#dstack.api.RunCollection.submit","title":"submit(configuration, configuration_path=None, repo=None, backends=None, regions=None, instance_types=None, resources=None, spot_policy=None, retry_policy=None, max_duration=None, max_price=None, working_dir=None, run_name=None, reserve_ports=True)","text":"Submit a run
Parameters:
Name Type Description Defaultconfiguration Union[Task, Service] A run configuration.
requiredconfiguration_path Optional[str] The path to the configuration file, relative to the root directory of the repo.
None repo Union[LocalRepo, RemoteRepo, VirtualRepo] A repo to mount to the run.
None backends Optional[List[BackendType]] A list of allowed backend for provisioning.
None regions Optional[List[str]] A list of cloud regions for provisioning.
None resources Optional[ResourcesSpec] The requirements to run the configuration. Overrides the configuration's resources.
None spot_policy Optional[SpotPolicy] A spot policy for provisioning.
None retry_policy RetryPolicy A retry policy.
None max_duration Optional[Union[int, str]] The max instance running duration in seconds.
None max_price Optional[float] The max instance price in dollars per hour for provisioning.
None working_dir Optional[str] A working directory relative to the repo root directory
None run_name Optional[str] A desired name of the run. Must be unique in the project. If not specified, a random name is assigned.
None reserve_ports bool Whether local ports should be reserved in advance.
True Returns:
Type DescriptionRun submitted run
"},{"location":"docs/reference/api/python/#dstack.api.Client.repos","title":"dstack.api.RepoCollection","text":"Operations with repos
"},{"location":"docs/reference/api/python/#dstack.api.RepoCollection.init","title":"init(repo, git_identity_file=None, oauth_token=None)","text":"Initializes the repo and configures its credentials in the project. Must be invoked before mounting the repo to a run.
Example:
repo=RemoteRepo.from_url(\n repo_url=\"https://github.com/dstackai/dstack-examples\",\n repo_branch=\"main\",\n)\nclient.repos.init(repo)\nBy default, it uses the default Git credentials configured on the machine. You can override these credentials via the git_identity_file or oauth_token arguments of the init method.
Once the repo is initialized, you can pass the repo object to the run:
run = client.runs.submit(\n configuration=...,\n repo=repo,\n)\nParameters:
Name Type Description Defaultrepo Repo The repo to initialize.
requiredgit_identity_file Optional[PathLike] The private SSH key path for accessing the remote repo.
None oauth_token Optional[str] The GitHub OAuth token to access the remote repo.
None"},{"location":"docs/reference/api/python/#dstack.api.Task","title":"dstack.api.Task","text":""},{"location":"docs/reference/api/python/#nodes","title":"nodes - (Optional) Number of nodes. Defaults to 1.","text":""},{"location":"docs/reference/api/python/#name","title":"name - (Optional) The run name.","text":""},{"location":"docs/reference/api/python/#image","title":"image - (Optional) The name of the Docker image to run.","text":""},{"location":"docs/reference/api/python/#entrypoint","title":"entrypoint - (Optional) The Docker entrypoint.","text":""},{"location":"docs/reference/api/python/#working_dir","title":"working_dir - (Optional) The path to the working directory inside the container. It's specified relative to the repository directory (/workflow) and should be inside it. Defaults to \".\" .","text":""},{"location":"docs/reference/api/python/#home_dir","title":"home_dir - (Optional) The absolute path to the home directory inside the container. Defaults to /root. Defaults to /root.","text":""},{"location":"docs/reference/api/python/#_registry_auth","title":"registry_auth - (Optional) Credentials for pulling a private Docker image.","text":""},{"location":"docs/reference/api/python/#python","title":"python - (Optional) The major version of Python. Mutually exclusive with image.","text":""},{"location":"docs/reference/api/python/#env","title":"env - (Optional) The mapping or the list of environment variables.","text":""},{"location":"docs/reference/api/python/#setup","title":"setup - (Optional) The bash commands to run on the boot.","text":""},{"location":"docs/reference/api/python/#_resources","title":"resources - (Optional) The resources requirements to run the configuration.","text":""},{"location":"docs/reference/api/python/#_volumes","title":"volumes - (Optional) The volumes mount points.","text":""},{"location":"docs/reference/api/python/#ports","title":"ports - (Optional) Port numbers/mapping to expose.","text":""},{"location":"docs/reference/api/python/#commands","title":"commands - (Optional) The bash commands to run.","text":""},{"location":"docs/reference/api/python/#backends","title":"backends - (Optional) The backends to consider for provisioning (e.g., [aws, gcp]).","text":""},{"location":"docs/reference/api/python/#regions","title":"regions - (Optional) The regions to consider for provisioning (e.g., [eu-west-1, us-west4, westeurope]).","text":""},{"location":"docs/reference/api/python/#instance_types","title":"instance_types - (Optional) The cloud-specific instance types to consider for provisioning (e.g., [p3.8xlarge, n1-standard-4]).","text":""},{"location":"docs/reference/api/python/#spot_policy","title":"spot_policy - (Optional) The policy for provisioning spot or on-demand instances: spot, on-demand, or auto.","text":""},{"location":"docs/reference/api/python/#_retry","title":"retry - (Optional) The policy for resubmitting the run. Defaults to false.","text":""},{"location":"docs/reference/api/python/#_retry_policy","title":"retry_policy - (Optional) The policy for resubmitting the run. Deprecated in favor of retry.","text":""},{"location":"docs/reference/api/python/#max_duration","title":"max_duration - (Optional) The maximum duration of a run (e.g., 2h, 1d, etc). After it elapses, the run is forced to stop. Defaults to off.","text":""},{"location":"docs/reference/api/python/#max_price","title":"max_price - (Optional) The maximum instance price per hour, in dollars.","text":""},{"location":"docs/reference/api/python/#pool_name","title":"pool_name - (Optional) The name of the pool. If not set, dstack will use the default name.","text":""},{"location":"docs/reference/api/python/#instance_name","title":"instance_name - (Optional) The name of the instance.","text":""},{"location":"docs/reference/api/python/#creation_policy","title":"creation_policy - (Optional) The policy for using instances from the pool. Defaults to reuse-or-create.","text":""},{"location":"docs/reference/api/python/#termination_policy","title":"termination_policy - (Optional) The policy for instance termination. Defaults to destroy-after-idle.","text":""},{"location":"docs/reference/api/python/#termination_idle_time","title":"termination_idle_time - (Optional) Time to wait before destroying the idle instance. Defaults to 5m for dstack run and to 3d for dstack pool add.","text":""},{"location":"docs/reference/api/python/#dstack.api.Service","title":"dstack.api.Service","text":""},{"location":"docs/reference/api/python/#port","title":"port - The port, that application listens on or the mapping.","text":""},{"location":"docs/reference/api/python/#model","title":"model - (Optional) Mapping of the model for the OpenAI-compatible endpoint.","text":""},{"location":"docs/reference/api/python/#https","title":"https - (Optional) Enable HTTPS. Defaults to True.","text":""},{"location":"docs/reference/api/python/#auth","title":"auth - (Optional) Enable the authorization. Defaults to True.","text":""},{"location":"docs/reference/api/python/#replicas","title":"replicas - (Optional) The number of replicas. Can be a number (e.g. 2) or a range (0..4 or 1..8). If it's a range, the scaling property is required. Defaults to 1.","text":""},{"location":"docs/reference/api/python/#_scaling","title":"scaling - (Optional) The auto-scaling rules. Required if replicas is set to a range.","text":""},{"location":"docs/reference/api/python/#name","title":"name - (Optional) The run name.","text":""},{"location":"docs/reference/api/python/#image","title":"image - (Optional) The name of the Docker image to run.","text":""},{"location":"docs/reference/api/python/#entrypoint","title":"entrypoint - (Optional) The Docker entrypoint.","text":""},{"location":"docs/reference/api/python/#working_dir","title":"working_dir - (Optional) The path to the working directory inside the container. It's specified relative to the repository directory (/workflow) and should be inside it. Defaults to \".\" .","text":""},{"location":"docs/reference/api/python/#home_dir","title":"home_dir - (Optional) The absolute path to the home directory inside the container. Defaults to /root. Defaults to /root.","text":""},{"location":"docs/reference/api/python/#_registry_auth","title":"registry_auth - (Optional) Credentials for pulling a private Docker image.","text":""},{"location":"docs/reference/api/python/#python","title":"python - (Optional) The major version of Python. Mutually exclusive with image.","text":""},{"location":"docs/reference/api/python/#env","title":"env - (Optional) The mapping or the list of environment variables.","text":""},{"location":"docs/reference/api/python/#setup","title":"setup - (Optional) The bash commands to run on the boot.","text":""},{"location":"docs/reference/api/python/#_resources","title":"resources - (Optional) The resources requirements to run the configuration.","text":""},{"location":"docs/reference/api/python/#_volumes","title":"volumes - (Optional) The volumes mount points.","text":""},{"location":"docs/reference/api/python/#commands","title":"commands - (Optional) The bash commands to run.","text":""},{"location":"docs/reference/api/python/#backends","title":"backends - (Optional) The backends to consider for provisioning (e.g., [aws, gcp]).","text":""},{"location":"docs/reference/api/python/#regions","title":"regions - (Optional) The regions to consider for provisioning (e.g., [eu-west-1, us-west4, westeurope]).","text":""},{"location":"docs/reference/api/python/#instance_types","title":"instance_types - (Optional) The cloud-specific instance types to consider for provisioning (e.g., [p3.8xlarge, n1-standard-4]).","text":""},{"location":"docs/reference/api/python/#spot_policy","title":"spot_policy - (Optional) The policy for provisioning spot or on-demand instances: spot, on-demand, or auto.","text":""},{"location":"docs/reference/api/python/#_retry","title":"retry - (Optional) The policy for resubmitting the run. Defaults to false.","text":""},{"location":"docs/reference/api/python/#_retry_policy","title":"retry_policy - (Optional) The policy for resubmitting the run. Deprecated in favor of retry.","text":""},{"location":"docs/reference/api/python/#max_duration","title":"max_duration - (Optional) The maximum duration of a run (e.g., 2h, 1d, etc). After it elapses, the run is forced to stop. Defaults to off.","text":""},{"location":"docs/reference/api/python/#max_price","title":"max_price - (Optional) The maximum instance price per hour, in dollars.","text":""},{"location":"docs/reference/api/python/#pool_name","title":"pool_name - (Optional) The name of the pool. If not set, dstack will use the default name.","text":""},{"location":"docs/reference/api/python/#instance_name","title":"instance_name - (Optional) The name of the instance.","text":""},{"location":"docs/reference/api/python/#creation_policy","title":"creation_policy - (Optional) The policy for using instances from the pool. Defaults to reuse-or-create.","text":""},{"location":"docs/reference/api/python/#termination_policy","title":"termination_policy - (Optional) The policy for instance termination. Defaults to destroy-after-idle.","text":""},{"location":"docs/reference/api/python/#termination_idle_time","title":"termination_idle_time - (Optional) Time to wait before destroying the idle instance. Defaults to 5m for dstack run and to 3d for dstack pool add.","text":""},{"location":"docs/reference/api/python/#dstack.api.Run","title":"dstack.api.Run","text":"Attributes:
Name Type Descriptionname str run name
ports Optional[Dict[int, int]] ports mapping, if run is attached
backend Optional[BackendType] backend type
status RunStatus run status
hostname str instance hostname
"},{"location":"docs/reference/api/python/#dstack.api.Run.attach","title":"attach(ssh_identity_file=None)","text":"Establish an SSH tunnel to the instance and update SSH config
Parameters:
Name Type Description Defaultssh_identity_file Optional[PathLike] SSH keypair to access instances
None Raises:
Type DescriptionPortUsedError If ports are in use or the run is attached by another process.
"},{"location":"docs/reference/api/python/#dstack.api.Run.detach","title":"detach()","text":"Stop the SSH tunnel to the instance and update SSH config
"},{"location":"docs/reference/api/python/#dstack.api.Run.logs","title":"logs(start_time=None, diagnose=False, replica_num=0, job_num=0)","text":"Iterate through run's log messages
Parameters:
Name Type Description Defaultstart_time Optional[datetime] minimal log timestamp
None diagnose bool return runner logs if True
False Yields:
Type DescriptionIterable[bytes] log messages
"},{"location":"docs/reference/api/python/#dstack.api.Run.refresh","title":"refresh()","text":"Get up-to-date run info
"},{"location":"docs/reference/api/python/#dstack.api.Run.stop","title":"stop(abort=False)","text":"Terminate the instance and detach
Parameters:
Name Type Description Defaultabort bool gracefully stop the run if False
False"},{"location":"docs/reference/api/python/#dstack.api.Resources","title":"dstack.api.Resources","text":""},{"location":"docs/reference/api/python/#cpu","title":"cpu - (Optional) The number of CPU cores. Defaults to 2...","text":""},{"location":"docs/reference/api/python/#memory","title":"memory - (Optional) The RAM size (e.g., 8GB). Defaults to 8GB...","text":""},{"location":"docs/reference/api/python/#shm_size","title":"shm_size - (Optional) The size of shared memory (e.g., 8GB). If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure this.","text":""},{"location":"docs/reference/api/python/#_gpu","title":"gpu - (Optional) The GPU requirements.","text":""},{"location":"docs/reference/api/python/#_disk","title":"disk - (Optional) The disk resources.","text":""},{"location":"docs/reference/api/python/#dstack.api.GPU","title":"dstack.api.GPU","text":""},{"location":"docs/reference/api/python/#name","title":"name - (Optional) The name of the GPU (e.g., A100 or H100).","text":""},{"location":"docs/reference/api/python/#count","title":"count - (Optional) The number of GPUs. Defaults to 1.","text":""},{"location":"docs/reference/api/python/#memory","title":"memory - (Optional) The RAM size (e.g., 16GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).","text":""},{"location":"docs/reference/api/python/#total_memory","title":"total_memory - (Optional) The total RAM size (e.g., 32GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).","text":""},{"location":"docs/reference/api/python/#compute_capability","title":"compute_capability - (Optional) The minimum compute capability of the GPU (e.g., 7.5).","text":""},{"location":"docs/reference/api/python/#dstack.api.Disk","title":"dstack.api.Disk","text":""},{"location":"docs/reference/api/python/#size","title":"size - Disk size.","text":""},{"location":"docs/reference/api/python/#dstack.api.LocalRepo","title":"dstack.api.LocalRepo","text":"Creates an instance of a local repo from a local path.
Example:
run = client.runs.submit(\n configuration=...,\n repo=LocalRepo.from_dir(\".\"), # Mount the current folder to the run\n)\nfrom_dir(repo_dir) staticmethod","text":"Creates an instance of a local repo from a local path.
Parameters:
Name Type Description Defaultrepo_dir PathLike The path to a local folder
requiredReturns:
Type DescriptionLocalRepo A local repo instance
"},{"location":"docs/reference/api/python/#dstack.api.RemoteRepo","title":"dstack.api.RemoteRepo","text":"Creates an instance of a remote Git repo for mounting to a submitted run.
Using a locally checked-out remote Git repo:
repo=RemoteRepo.from_dir(repo_dir=\".\")\nUsing a remote Git repo by a URL:
repo=RemoteRepo.from_url(\n repo_url=\"https://github.com/dstackai/dstack-examples\",\n repo_branch=\"main\"\n)\nInitialize the repo before mounting it.
client.repos.init(repo)\nBy default, it uses the default Git credentials configured on the machine. You can override these credentials via the git_identity_file or oauth_token arguments of the init method.
Finally, you can pass the repo object to the run:
run = client.runs.submit(\n configuration=...,\n repo=repo,\n)\nfrom_dir(repo_dir) staticmethod","text":"Creates an instance of a remote repo from a local path.
Parameters:
Name Type Description Defaultrepo_dir PathLike The path to a local folder
requiredReturns:
Type DescriptionRemoteRepo A remote repo instance
"},{"location":"docs/reference/api/python/#dstack.api.RemoteRepo.from_url","title":"from_url(repo_url, repo_branch=None, repo_hash=None) staticmethod","text":"Creates an instance of a remote repo from a URL.
Parameters:
Name Type Description Defaultrepo_url str The URL of a remote Git repo
requiredrepo_branch Optional[str] The name of the remote branch. Must be specified if hash is not specified.
None repo_hash Optional[str] The hash of the revision. Must be specified if branch is not specified.
None Returns:
Type DescriptionRemoteRepo A remote repo instance
"},{"location":"docs/reference/api/python/#dstack.api.VirtualRepo","title":"dstack.api.VirtualRepo","text":"Allows mounting a repo created programmatically.
Example:
virtual_repo = VirtualRepo(repo_id=\"some-unique-repo-id\")\nvirtual_repo.add_file_from_package(package=some_package, path=\"requirements.txt\")\nvirtual_repo.add_file_from_package(package=some_package, path=\"train.py\")\n\nrun = client.runs.submit(\n configuration=...,\n repo=virtual_repo,\n)\nAttributes:
Name Type Descriptionrepo_id A unique identifier of the repo
"},{"location":"docs/reference/api/python/#dstack.api.VirtualRepo.add_file","title":"add_file(path, content)","text":"Adds a given file to the repo.
Attributes:
Name Type Descriptionpath str The path inside the repo to add the file.
content bytes The contents of the file.
"},{"location":"docs/reference/api/python/#dstack.api.VirtualRepo.add_file_from_package","title":"add_file_from_package(package, path)","text":"Includes a file from a given package to the repo.
Attributes:
Name Type Descriptionpackage Union[ModuleType, str] A package to include the file from.
path str The path to the file to include to the repo. Must be relative to the package directory.
"},{"location":"docs/reference/api/python/#dstack.api.RegistryAuth","title":"dstack.api.RegistryAuth","text":""},{"location":"docs/reference/api/python/#username","title":"username - The username.","text":""},{"location":"docs/reference/api/python/#password","title":"password - The password or access token.","text":""},{"location":"docs/reference/api/python/#dstack.api.Scaling","title":"dstack.api.Scaling","text":""},{"location":"docs/reference/api/python/#metric","title":"metric - The target metric to track. Currently, the only supported value is rps (meaning requests per second).","text":""},{"location":"docs/reference/api/python/#target","title":"target - The target value of the metric. The number of replicas is calculated based on this number and automatically adjusts (scales up or down) as this metric changes.","text":""},{"location":"docs/reference/api/python/#scale_up_delay","title":"scale_up_delay - (Optional) The delay in seconds before scaling up. Defaults to 300.","text":""},{"location":"docs/reference/api/python/#scale_down_delay","title":"scale_down_delay - (Optional) The delay in seconds before scaling down. Defaults to 600.","text":""},{"location":"docs/reference/api/python/#dstack.api.BackendType","title":"dstack.api.BackendType","text":"Attributes:
Name Type DescriptionAWS BackendType Amazon Web Services
AZURE BackendType Microsoft Azure
CUDO BackendType Cudo
DSTACK BackendType dstack Sky
GCP BackendType Google Cloud Platform
DATACRUNCH BackendType DataCrunch
KUBERNETES BackendType Kubernetes
LAMBDA BackendType Lambda Cloud
RUNPOD BackendType Runpod Cloud
TENSORDOCK BackendType TensorDock Marketplace
VASTAI BackendType Vast.ai Marketplace
"},{"location":"docs/reference/api/rest/","title":"REST API","text":""},{"location":"docs/reference/cli/","title":"CLI","text":""},{"location":"docs/reference/cli/#commands","title":"Commands","text":""},{"location":"docs/reference/cli/#dstack-server","title":"dstack server","text":"This command starts the dstack server.
$ dstack server --help\nUsage: dstack server [-h] [--host HOST] [-p PORT] [-l LOG_LEVEL] [--default]\n [--no-default] [--token TOKEN]\n\nOptions:\n -h, --help Show this help message and exit\n --host HOST Bind socket to this host. Defaults to 127.0.0.1\n -p, --port PORT Bind socket to this port. Defaults to 3000.\n -l, --log-level LOG_LEVEL\n Server logging level. Defaults to INFO.\n --default Update the default project configuration\n --no-default Do not update the default project configuration\n --token TOKEN The admin user token\nThis command must be called inside a folder before you can run dstack apply.
Git credentials
If the current folder is a remote Git repository, dstack init ensures that dstack can access it. By default, the command uses the remote repo's default Git credentials. These can be overridden with --git-identity (private SSH key) or --token (OAuth token).
$ dstack init --help\nUsage: dstack init [-h] [--project PROJECT] [-t OAUTH_TOKEN]\n [--git-identity SSH_PRIVATE_KEY]\n [--ssh-identity SSH_PRIVATE_KEY] [--local]\n\nOptions:\n -h, --help Show this help message and exit\n --project PROJECT The name of the project\n -t, --token OAUTH_TOKEN\n An authentication token for Git\n --git-identity SSH_PRIVATE_KEY\n The private SSH key path to access the remote repo\n --ssh-identity SSH_PRIVATE_KEY\n The private SSH key path for SSH tunneling\n --local Do not use git\nUser SSH key
By default, dstack uses its own SSH key to access instances (~/.dstack/ssh/id_rsa). It is possible to override this key via the --ssh-identity argument.
This command applies a given configuration. If a resource does not exist, dstack apply creates the resource. If a resource exists, dstack apply updates the resource in-place or re-creates the resource if the update is not possible.
$ dstack apply --help\nUsage: dstack apply [--project NAME] [-h [TYPE]] [-f FILE] [--force] [-y]\n\nOptions:\n --project NAME The name of the project. Defaults to $DSTACK_PROJECT\n -h, --help [TYPE] Show this help message and exit.\n -f, --file FILE The path to the configuration file. Defaults to\n $PWD/.dstack.yml\n --force Force apply when no changes detected\n -y, --yes Do not ask for confirmation\nThis command deletes the resources defined by a given configuration.
$ dstack delete --help\nUsage: dstack delete [-h] [--project NAME] [-f FILE] [-y]\n\nOptions:\n -h, --help Show this help message and exit\n --project NAME The name of the project. Defaults to $DSTACK_PROJECT\n -f, --file FILE The path to the configuration file. Defaults to\n $PWD/.dstack.yml\n -y, --yes Do not ask for confirmation\nNOTE:
The dstack delete command currently supports only gateway configurations. Support for other configuration types is coming soon.
This command shows the status of runs.
$ dstack ps --help\nUsage: dstack ps [-h] [--project NAME] [-a] [-v] [-w]\n\nOptions:\n -h, --help Show this help message and exit\n --project NAME The name of the project. Defaults to $DSTACK_PROJECT\n -a, --all Show all runs. By default, it only shows unfinished runs or\n the last finished.\n -v, --verbose Show more information about runs\n -w, --watch Watch statuses of runs in realtime\nThis command stops run(s) within the current repository.
$ dstack stop --help\nUsage: dstack stop [-h] [--project NAME] [-x] [-y] run_name\n\nPositional Arguments:\n run_name\n\nOptions:\n -h, --help Show this help message and exit\n --project NAME The name of the project. Defaults to $DSTACK_PROJECT\n -x, --abort\n -y, --yes\nThis command shows the output of a given run within the current repository.
$ dstack logs --help\nUsage: dstack logs [-h] [--project NAME] [-d] [-a]\n [--ssh-identity SSH_PRIVATE_KEY] [--replica REPLICA]\n [--job JOB]\n run_name\n\nPositional Arguments:\n run_name\n\nOptions:\n -h, --help Show this help message and exit\n --project NAME The name of the project. Defaults to $DSTACK_PROJECT\n -d, --diagnose\n -a, --attach Set up an SSH tunnel, and print logs as they follow.\n --ssh-identity SSH_PRIVATE_KEY\n The private SSH key path for SSH tunneling\n --replica REPLICA The relica number. Defaults to 0.\n --job JOB The job number inside the replica. Defaults to 0.\nBoth the CLI and API need to be configured with the server address, user token, and project name via ~/.dstack/config.yml.
At startup, the server automatically configures CLI and API with the server address, user token, and the default project name (main). This configuration is stored via ~/.dstack/config.yml.
To use CLI and API on different machines or projects, use the dstack config command.
$ dstack config --help\nUsage: dstack config [-h] [--project PROJECT] [--url URL] [--token TOKEN]\n [--default] [--remove] [--no-default]\n\nOptions:\n -h, --help Show this help message and exit\n --project PROJECT The name of the project to configure\n --url URL Server url\n --token TOKEN User token\n --default Set the project as default. It will be used when\n --project is omitted in commands.\n --remove Delete project configuration\n --no-default Do not prompt to set the project as default\nFleets enable efficient provisioning and management of clusters and instances.
"},{"location":"docs/reference/cli/#dstack-fleet-list","title":"dstack fleet list","text":"The dstack fleet list command displays fleets and instances.
$ dstack fleet list --help\nUsage: dstack fleet list [-h] [-w] [-v]\n\nOptions:\n -h, --help show this help message and exit\n -w, --watch Update listing in realtime\n -v, --verbose Show more information\nThe dstack fleet delete deletes fleets and instances. Cloud instances are terminated upon deletion.
$ dstack fleet delete --help\nUsage: dstack fleet delete [-h] [-i INSTANCE_NUM] [-y] name\n\nPositional Arguments:\n name The name of the fleet\n\nOptions:\n -h, --help show this help message and exit\n -i, --instance INSTANCE_NUM\n The instances to delete\n -y, --yes Don't ask for confirmation\nA gateway is required for running services. It handles ingress traffic, authorization, domain mapping, model mapping for the OpenAI-compatible endpoint, and so on.
"},{"location":"docs/reference/cli/#dstack-gateway-list","title":"dstack gateway list","text":"The dstack gateway list command displays the names and addresses of the gateways configured in the project.
$ dstack gateway list --help\nUsage: dstack gateway list [-h] [-w] [-v]\n\nOptions:\n -h, --help show this help message and exit\n -w, --watch Update listing in realtime\n -v, --verbose Show more information\nThe dstack gateway create command creates a new gateway instance in the project.
$ dstack gateway create --help\nUsage: dstack gateway create [-h] --backend {aws,azure,gcp,kubernetes}\n --region REGION [--set-default] [--name NAME]\n --domain DOMAIN\n\nOptions:\n -h, --help show this help message and exit\n --backend {aws,azure,gcp,kubernetes}\n --region REGION\n --set-default Set as default gateway for the project\n --name NAME Set a custom name for the gateway\n --domain DOMAIN Set the domain for the gateway\nThe dstack gateway delete command deletes the specified gateway.
$ dstack gateway delete --help\nUsage: dstack gateway delete [-h] [-y] name\n\nPositional Arguments:\n name The name of the gateway\n\nOptions:\n -h, --help show this help message and exit\n -y, --yes Don't ask for confirmation\nThe dstack gateway update command updates the specified gateway.
$ dstack gateway update --help\nUsage: dstack gateway update [-h] [--set-default] [--domain DOMAIN] name\n\nPositional Arguments:\n name The name of the gateway\n\nOptions:\n -h, --help show this help message and exit\n --set-default Set it the default gateway for the project\n --domain DOMAIN Set the domain for the gateway\nThe volumes commands.
"},{"location":"docs/reference/cli/#dstack-volume-list","title":"dstack volume list","text":"The dstack volume list command lists volumes.
$ dstack volume list --help\nUsage: dstack volume list [-h] [-w] [-v]\n\nOptions:\n -h, --help show this help message and exit\n -w, --watch Update listing in realtime\n -v, --verbose Show more information\nThe dstack volume delete command deletes volumes.
$ dstack volume delete --help\nUsage: dstack volume delete [-h] [-y] name\n\nPositional Arguments:\n name The name of the volume\n\nOptions:\n -h, --help show this help message and exit\n -y, --yes Don't ask for confirmation\nThis command runs a given configuration.
Deprecation
dstack run is deprecated in favor of dstack apply.
$ dstack run . --help\nUsage: dstack run [--project NAME] [-h [TYPE]] [-f FILE] [-y] [-n RUN_NAME]\n [-d] [--max-offers MAX_OFFERS] [-e KEY=VALUE] [--gpu SPEC]\n [--disk RANGE] [--profile NAME] [--max-price PRICE]\n [--max-duration DURATION] [-b NAME] [-r NAME]\n [--instance-type NAME]\n [--pool POOL_NAME | --reuse | --dont-destroy | --idle-duration IDLE_DURATION | --instance NAME]\n [--spot | --on-demand | --spot-auto | --spot-policy POLICY]\n [--retry | --no-retry | --retry-duration DURATION]\n working_dir\n\nPositional Arguments:\n working_dir\n\nOptions:\n --project NAME The name of the project. Defaults to $DSTACK_PROJECT\n -h, --help [TYPE] Show this help message and exit. TYPE is one of task,\n dev-environment, service\n -f, --file FILE The path to the configuration file. Defaults to\n $PWD/.dstack.yml\n -y, --yes Do not ask for confirmation\n -n, --name RUN_NAME The name of the run. If not specified, a random name\n is assigned\n -d, --detach Do not poll logs and run status\n --max-offers MAX_OFFERS\n Number of offers to show in the run plan\n -e, --env KEY=VALUE Environment variables\n --gpu SPEC Request GPU for the run. The format is\n NAME:COUNT:MEMORY (all parts are optional)\n --disk RANGE Request the size range of disk for the run. Example\n --disk 100GB...\n\nProfile:\n --profile NAME The name of the profile. Defaults to $DSTACK_PROFILE\n --max-price PRICE The maximum price per hour, in dollars\n --max-duration DURATION\n The maximum duration of the run\n -b, --backend NAME The backends that will be tried for provisioning\n -r, --region NAME The regions that will be tried for provisioning\n --instance-type NAME The cloud-specific instance types that will be tried\n for provisioning\n\nPools:\n --pool POOL_NAME The name of the pool. If not set, the default pool\n will be used\n --reuse Reuse instance from pool\n --dont-destroy Do not destroy instance after the run is finished\n --idle-duration IDLE_DURATION\n Time to wait before destroying the idle instance\n --instance NAME Reuse instance from pool with name NAME\n\nSpot Policy:\n --spot Consider only spot instances\n --on-demand Consider only on-demand instances\n --spot-auto Consider both spot and on-demand instances\n --spot-policy POLICY One of spot, on-demand, auto\n\nRetry Policy:\n --retry\n --no-retry\n --retry-duration DURATION\nWhen running anything via CLI, dstack uses the exact version of code from your project directory.
If there are large files, consider creating a .gitignore file to exclude them for better performance.
Pools allow for managing the lifecycle of instances and reusing them across runs. The default pool is created automatically.
Deprecation
Pools are deprecated in favor of fleets and will be removed in 0.19.0.
"},{"location":"docs/reference/cli/#dstack-pool-add","title":"dstack pool add","text":"The dstack pool add command provisions a cloud instance and adds it to a pool. If no pool name is specified, the instance goes to the default pool.
$ dstack pool add --help\nUsage: dstack pool add [-h] [-y] [--profile NAME] [--max-price PRICE]\n [-b NAME] [-r NAME] [--instance-type NAME]\n [--pool POOL_NAME] [--reuse] [--dont-destroy]\n [--idle-duration IDLE_DURATION]\n [--spot | --on-demand | --spot-auto | --spot-policy POLICY]\n [--retry | --no-retry | --retry-duration DURATION]\n [--cpu SPEC] [--memory SIZE] [--shared-memory SIZE]\n [--gpu SPEC] [--disk SIZE]\n\nOptions:\n -h, --help show this help message and exit\n -y, --yes Don't ask for confirmation\n --pool POOL_NAME The name of the pool. If not set, the default pool\n will be used\n --reuse Reuse instance from pool\n --dont-destroy Do not destroy instance after the run is finished\n --idle-duration IDLE_DURATION\n Time to wait before destroying the idle instance\n\nProfile:\n --profile NAME The name of the profile. Defaults to $DSTACK_PROFILE\n --max-price PRICE The maximum price per hour, in dollars\n -b, --backend NAME The backends that will be tried for provisioning\n -r, --region NAME The regions that will be tried for provisioning\n --instance-type NAME The cloud-specific instance types that will be tried\n for provisioning\n\nSpot Policy:\n --spot Consider only spot instances\n --on-demand Consider only on-demand instances\n --spot-auto Consider both spot and on-demand instances\n --spot-policy POLICY One of spot, on-demand, auto\n\nRetry Policy:\n --retry\n --no-retry\n --retry-duration DURATION\n\nResources:\n --cpu SPEC Request the CPU count. Default: 2..\n --memory SIZE Request the size of RAM. The format is SIZE:MB|GB|TB.\n Default: 8GB..\n --shared-memory SIZE Request the size of Shared Memory. The format is\n SIZE:MB|GB|TB.\n --gpu SPEC Request GPU for the run. The format is\n NAME:COUNT:MEMORY (all parts are optional)\n --disk SIZE Request the size of disk for the run. Example --disk\n 100GB...\nThe dstack pool add-ssh command adds an existing remote instance to a pool. If no pool name is specified, the instance goes to the default pool.
$ dstack pool add-ssh --help\nUsage: dstack pool add-ssh [-h] -i SSH_PRIVATE_KEY [-p SSH_PORT]\n [-l LOGIN_NAME] [--region REGION]\n [--pool POOL_NAME] [--name INSTANCE_NAME]\n [--network NETWORK]\n destination\n\nPositional Arguments:\n destination\n\nOptions:\n -h, --help show this help message and exit\n -i SSH_PRIVATE_KEY The private SSH key path for SSH\n -p SSH_PORT SSH port to connect\n -l LOGIN_NAME User to login\n --region REGION Host region\n --pool POOL_NAME Pool name\n --name INSTANCE_NAME Set the name of the instance\n --network NETWORK Network address for multinode setup. Format <ip\n address>/<netmask>\nThe dstack pool ps command lists all active instances of a pool. If no pool name is specified, default pool instances are displayed.
$ dstack pool ps --help\nUsage: dstack pool ps [-h] [--pool POOL_NAME] [-w]\n\nShow instances in the pool\n\nOptions:\n -h, --help show this help message and exit\n --pool POOL_NAME The name of the pool. If not set, the default pool will be\n used\n -w, --watch Watch instances in realtime\nThe dstack pool rm command removes an instance from a pool. Cloud instances are terminated upon removal.
$ dstack pool rm --help\nUsage: dstack pool rm [-h] [--pool POOL_NAME] [--force] [-y] instance_name\n\nPositional Arguments:\n instance_name The name of the instance\n\nOptions:\n -h, --help show this help message and exit\n --pool POOL_NAME The name of the pool. If not set, the default pool will be\n used\n --force The name of the instance\n -y, --yes Don't ask for confirmation\nThe dstack pool create command creates a new pool.
$ dstack pool create --help\nUsage: dstack pool create [-h] -n POOL_NAME\n\nOptions:\n -h, --help show this help message and exit\n -n, --name POOL_NAME The name of the pool\nThe dstack pool list command lists all existing pools.
$ dstack pool list --help\nUsage: dstack pool list [-h] [-v VERBOSE]\n\nList available pools\n\nOptions:\n -h, --help show this help message and exit\n -v, --verbose VERBOSE\n Show more information\nThe dstack pool set-default command sets the project's default pool.
$ dstack pool set-default --help\nUsage: dstack pool set-default [-h] --pool POOL_NAME\n\nOptions:\n -h, --help show this help message and exit\n --pool POOL_NAME The name of the pool\nThe dstack pool delete command deletes a specified pool.
$ dstack pool delete --help\nUsage: dstack pool delete [-h] -n POOL_NAME\n\nOptions:\n -h, --help show this help message and exit\n -n, --name POOL_NAME The name of the pool\nDSTACK_CLI_LOG_LEVEL \u2013 (Optional) Configures CLI logging level. Defaults to INFO.DSTACK_SERVER_LOG_LEVEL \u2013 (Optional) Has the same effect as --log-level. Defaults to INFO.DSTACK_SERVER_HOST \u2013 (Optional) Has the same effect as --host. Defaults to 127.0.0.1.DSTACK_SERVER_PORT \u2013 (Optional) Has the same effect as --port. Defaults to 3000.DSTACK_SERVER_ADMIN_TOKEN \u2013 (Optional) Has the same effect as --token. Defaults to None.DSTACK_DATABASE_URL \u2013 (Optional) The database URL to use instead of default SQLite. Currently dstack supports Postgres. Example: postgresql+asyncpg://myuser:mypassword@localhost:5432/mydatabase. Defaults to None.DSTACK_SERVER_DIR \u2013 (Optional) Sets path to store data and server configs. Defaults to ~/.dstack/server.DSTACK_SERVER_ROOT_LOG_LEVEL \u2013 (Optional) Sets root logger log level. Defaults to ERROR.DSTACK_SERVER_LOG_FORMAT \u2013 (Optional) Sets format of log output. Can be rich, standard, json.. Defaults to rich.DSTACK_SERVER_UVICORN_LOG_LEVEL \u2013 (Optional) Sets uvicorn logger log level. Defaults to ERROR.DSTACK_PROFILE \u2013 (Optional) Has the same effect as --profile. Defaults to None.DSTACK_PROJECT \u2013 (Optional) Has the same effect as --project. Defaults to None.DSTACK_RUNNER_VERSION \u2013 (Optional) Sets exact runner version for debug. Defaults to latest.DSTACK_DEFAULT_CREDS_DISABLED \u2013 (Optional) Disables default credentials detection if set. Defaults to None.DSTACK_LOCAL_BACKEND_ENABLED \u2013 (Optional) Enables local backend for debug if set. Defaults to None.The dev-environment configuration type allows running dev environments.
Configuration files must have a name ending with .dstack.yml (e.g., .dstack.yml or serve.dstack.yml are both acceptable) and can be located in the project's root directory or any nested folder. Any configuration can be run via dstack run.
If you don't specify image, dstack uses the default Docker image pre-configured with python, pip, conda (Miniforge), and essential CUDA drivers. The python property determines which default Docker image is used.
type: dev-environment\n\npython: \"3.11\"\n\nide: vscode\nnvcc
Note that the default Docker image doesn't bundle nvcc, which is required for building custom CUDA kernels. To install it, use conda install cuda.
type: dev-environment\n\nimage: ghcr.io/huggingface/text-generation-inference:latest\n\nide: vscode\nUse the registry_auth property to provide credentials for a private Docker registry.
type: dev-environment\n\nimage: ghcr.io/huggingface/text-generation-inference:latest\nregistry_auth:\n username: peterschmidt85\n password: ghp_e49HcZ9oYwBzUbcSk2080gXZOU2hiT9AeSR5\n\nide: vscode\nIf you specify memory size, you can either specify an explicit size (e.g. 24GB) or a range (e.g. 24GB.., or 24GB..80GB, or ..80GB).
type: dev-environment\n\nide: vscode\n\nresources:\n # 200GB or more RAM\n memory: 200GB..\n\n # 4 GPUs from 40GB to 80GB\n gpu: 40GB..80GB:4\n\n # Shared memory\n shm_size: 16GB\n\n disk: 500GB\nThe gpu property allows specifying not only memory size but also GPU names and their quantity. Examples: A100 (one A100), A10G,A100 (either A10G or A100), A100:80GB (one A100 of 80GB), A100:2 (two A100), 24GB..40GB:2 (two GPUs between 24GB and 40GB), A100:40GB:2 (two A100 GPUs of 40GB).
To use TPUs, specify its architecture prefixed by tpu- via the gpu property.
type: dev-environment\n\nide: vscode\n\nresources:\n gpu: tpu-v2-8\nCurrently, only 8 TPU cores can be specified, supporting single TPU device workloads. Multi-TPU support is coming soon.
Shared memoryIf you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure shm_size, e.g. set it to 16GB.
type: dev-environment\n\nenv:\n - HUGGING_FACE_HUB_TOKEN\n - HF_HUB_ENABLE_HF_TRANSFER=1\n\nide: vscode\nIf you don't assign a value to an environment variable (see HUGGING_FACE_HUB_TOKEN above), dstack will require the value to be passed via the CLI or set in the current process.
For instance, you can define environment variables in a .env file and utilize tools like direnv.
The following environment variables are available in any run and are passed by dstack by default:
DSTACK_RUN_NAME The name of the run DSTACK_REPO_ID The ID of the repo DSTACK_GPUS_NUM The total number of GPUs in the run"},{"location":"docs/reference/dstack.yml/dev-environment/#spot-policy","title":"Spot policy","text":"You can choose whether to use spot instances, on-demand instances, or any available type.
type: dev-environment\n\nide: vscode\n\nspot_policy: auto\nThe spot_policy accepts spot, on-demand, and auto. The default for dev environments is on-demand.
By default, dstack provisions instances in all configured backends. However, you can specify the list of backends:
type: dev-environment\n\nide: vscode\n\nbackends: [aws, gcp]\nBy default, dstack uses all configured regions. However, you can specify the list of regions:
type: dev-environment\n\nide: vscode\n\nregions: [eu-west-1, eu-west-2]\nVolumes allow you to persist data between runs. To attach a volume, simply specify its name using the volumes property and specify where to mount its contents:
type: dev-environment\n\nide: vscode\n\nvolumes:\n - name: my-new-volume\n path: /volume_data\nOnce you run this configuration, the contents of the volume will be attached to /volume_data inside the dev environment, and its contents will persist across runs.
Limitations
When you're running a dev environment, task, or service with dstack, it automatically mounts the project folder contents to /workflow (and sets that as the current working directory). Right now, dstack doesn't allow you to attach volumes to /workflow or any of its subdirectories.
The dev-environment configuration type supports many other options. See below.
ide - The IDE to run.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#version","title":"version - (Optional) The version of the IDE.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#init","title":"init - (Optional) The bash commands to run.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#name","title":"name - (Optional) The run name.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#image","title":"image - (Optional) The name of the Docker image to run.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#entrypoint","title":"entrypoint - (Optional) The Docker entrypoint.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#working_dir","title":"working_dir - (Optional) The path to the working directory inside the container. It's specified relative to the repository directory (/workflow) and should be inside it. Defaults to \".\" .","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#home_dir","title":"home_dir - (Optional) The absolute path to the home directory inside the container. Defaults to /root. Defaults to /root.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#_registry_auth","title":"registry_auth - (Optional) Credentials for pulling a private Docker image.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#python","title":"python - (Optional) The major version of Python. Mutually exclusive with image.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#env","title":"env - (Optional) The mapping or the list of environment variables.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#setup","title":"setup - (Optional) The bash commands to run on the boot.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#_resources","title":"resources - (Optional) The resources requirements to run the configuration.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#_volumes","title":"volumes - (Optional) The volumes mount points.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#ports","title":"ports - (Optional) Port numbers/mapping to expose.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#backends","title":"backends - (Optional) The backends to consider for provisioning (e.g., [aws, gcp]).","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#regions","title":"regions - (Optional) The regions to consider for provisioning (e.g., [eu-west-1, us-west4, westeurope]).","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#instance_types","title":"instance_types - (Optional) The cloud-specific instance types to consider for provisioning (e.g., [p3.8xlarge, n1-standard-4]).","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#spot_policy","title":"spot_policy - (Optional) The policy for provisioning spot or on-demand instances: spot, on-demand, or auto.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#_retry","title":"retry - (Optional) The policy for resubmitting the run. Defaults to false.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#_retry_policy","title":"retry_policy - (Optional) The policy for resubmitting the run. Deprecated in favor of retry.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#max_duration","title":"max_duration - (Optional) The maximum duration of a run (e.g., 2h, 1d, etc). After it elapses, the run is forced to stop. Defaults to off.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#max_price","title":"max_price - (Optional) The maximum instance price per hour, in dollars.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#pool_name","title":"pool_name - (Optional) The name of the pool. If not set, dstack will use the default name.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#instance_name","title":"instance_name - (Optional) The name of the instance.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#creation_policy","title":"creation_policy - (Optional) The policy for using instances from the pool. Defaults to reuse-or-create.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#termination_policy","title":"termination_policy - (Optional) The policy for instance termination. Defaults to destroy-after-idle.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#termination_idle_time","title":"termination_idle_time - (Optional) Time to wait before destroying the idle instance. Defaults to 5m for dstack run and to 3d for dstack pool add.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#resources","title":"resources","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#cpu","title":"cpu - (Optional) The number of CPU cores. Defaults to 2...","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#memory","title":"memory - (Optional) The RAM size (e.g., 8GB). Defaults to 8GB...","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#shm_size","title":"shm_size - (Optional) The size of shared memory (e.g., 8GB). If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure this.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#_gpu","title":"gpu - (Optional) The GPU requirements. Can be set to a number, a string (e.g. A100, 80GB:2, etc.), or an object.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#_disk","title":"disk - (Optional) The disk resources.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#resources-gpu","title":"resources.gpu","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#name","title":"name - (Optional) The GPU name or list of names.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#count","title":"count - (Optional) The number of GPUs. Defaults to 1.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#memory","title":"memory - (Optional) The RAM size (e.g., 16GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#total_memory","title":"total_memory - (Optional) The total RAM size (e.g., 32GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#compute_capability","title":"compute_capability - (Optional) The minimum compute capability of the GPU (e.g., 7.5).","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#resources-disk","title":"resources.disk","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#size","title":"size - The disk size. Can be a string (e.g., 100GB or 100GB..) or an object.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#registry_auth","title":"registry_auth","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#username","title":"username - The username.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#password","title":"password - The password or access token.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#volumes_1","title":"volumes","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#name","title":"name - The name of the volume to mount.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#path","title":"path - The container path to mount the volume at.","text":""},{"location":"docs/reference/dstack.yml/fleet/","title":"fleet","text":"The fleet configuration type allows creating and updating fleets.
Configuration files must have a name ending with .dstack.yml (e.g., .dstack.yml or fleet.dstack.yml are both acceptable) and can be located in the project's root directory or any nested folder. Any configuration can be applied via dstack apply.
type: fleet\nname: my-gcp-fleet\nnodes: 4\nplacement: cluster\nbackends: [gcp]\nresources:\n gpu: 1\ntype: fleet\nname: my-ssh-fleet\nssh_config:\n user: ubuntu\n identity_file: ~/.ssh/id_rsa\n hosts:\n - 3.255.177.51\n - 3.255.177.52\nname - (Optional) The fleet name.","text":""},{"location":"docs/reference/dstack.yml/fleet/#_ssh_config","title":"ssh_config - (Optional) The parameters for adding instances via SSH.","text":""},{"location":"docs/reference/dstack.yml/fleet/#nodes","title":"nodes - (Optional) The number of instances.","text":""},{"location":"docs/reference/dstack.yml/fleet/#placement","title":"placement - (Optional) The placement of instances: any or cluster.","text":""},{"location":"docs/reference/dstack.yml/fleet/#_resources","title":"resources - (Optional) The resources requirements.","text":""},{"location":"docs/reference/dstack.yml/fleet/#backends","title":"backends - (Optional) The backends to consider for provisioning (e.g., [aws, gcp]).","text":""},{"location":"docs/reference/dstack.yml/fleet/#regions","title":"regions - (Optional) The regions to consider for provisioning (e.g., [eu-west-1, us-west4, westeurope]).","text":""},{"location":"docs/reference/dstack.yml/fleet/#instance_types","title":"instance_types - (Optional) The cloud-specific instance types to consider for provisioning (e.g., [p3.8xlarge, n1-standard-4]).","text":""},{"location":"docs/reference/dstack.yml/fleet/#spot_policy","title":"spot_policy - (Optional) The policy for provisioning spot or on-demand instances: spot, on-demand, or auto.","text":""},{"location":"docs/reference/dstack.yml/fleet/#_retry","title":"retry - (Optional) The policy for provisioning retry. Defaults to false.","text":""},{"location":"docs/reference/dstack.yml/fleet/#max_price","title":"max_price - (Optional) The maximum instance price per hour, in dollars.","text":""},{"location":"docs/reference/dstack.yml/fleet/#termination_policy","title":"termination_policy - (Optional) The policy for instance termination. Defaults to destroy-after-idle.","text":""},{"location":"docs/reference/dstack.yml/fleet/#termination_idle_time","title":"termination_idle_time - (Optional) Time to wait before destroying idle instances. Defaults to 3d.","text":""},{"location":"docs/reference/dstack.yml/fleet/#ssh","title":"ssh","text":""},{"location":"docs/reference/dstack.yml/fleet/#user","title":"user - (Optional) The user to log in with on all hosts.","text":""},{"location":"docs/reference/dstack.yml/fleet/#port","title":"port - (Optional) The SSH port to connect to.","text":""},{"location":"docs/reference/dstack.yml/fleet/#identity_file","title":"identity_file - (Optional) The private key to use for all hosts.","text":""},{"location":"docs/reference/dstack.yml/fleet/#hosts","title":"hosts - The per host connection parameters: a hostname or an object that overrides default ssh parameters.","text":""},{"location":"docs/reference/dstack.yml/fleet/#network","title":"network - (Optional) The network address for cluster setup in the format <ip>/<netmask>.","text":""},{"location":"docs/reference/dstack.yml/fleet/#sshhostsn","title":"ssh.hosts[n]","text":""},{"location":"docs/reference/dstack.yml/fleet/#hostname","title":"hostname - The IP address or domain to connect to.","text":""},{"location":"docs/reference/dstack.yml/fleet/#port","title":"port - (Optional) The SSH port to connect to for this host.","text":""},{"location":"docs/reference/dstack.yml/fleet/#user","title":"user - (Optional) The user to log in with for this host.","text":""},{"location":"docs/reference/dstack.yml/fleet/#identity_file","title":"identity_file - (Optional) The private key to use for this host.","text":""},{"location":"docs/reference/dstack.yml/fleet/#resources","title":"resources","text":""},{"location":"docs/reference/dstack.yml/fleet/#cpu","title":"cpu - (Optional) The number of CPU cores. Defaults to 2...","text":""},{"location":"docs/reference/dstack.yml/fleet/#memory","title":"memory - (Optional) The RAM size (e.g., 8GB). Defaults to 8GB...","text":""},{"location":"docs/reference/dstack.yml/fleet/#shm_size","title":"shm_size - (Optional) The size of shared memory (e.g., 8GB). If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure this.","text":""},{"location":"docs/reference/dstack.yml/fleet/#_gpu","title":"gpu - (Optional) The GPU requirements. Can be set to a number, a string (e.g. A100, 80GB:2, etc.), or an object.","text":""},{"location":"docs/reference/dstack.yml/fleet/#_disk","title":"disk - (Optional) The disk resources.","text":""},{"location":"docs/reference/dstack.yml/fleet/#resources-gpu","title":"resouces.gpu","text":""},{"location":"docs/reference/dstack.yml/fleet/#name","title":"name - (Optional) The GPU name or list of names.","text":""},{"location":"docs/reference/dstack.yml/fleet/#count","title":"count - (Optional) The number of GPUs. Defaults to 1.","text":""},{"location":"docs/reference/dstack.yml/fleet/#memory","title":"memory - (Optional) The RAM size (e.g., 16GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).","text":""},{"location":"docs/reference/dstack.yml/fleet/#total_memory","title":"total_memory - (Optional) The total RAM size (e.g., 32GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).","text":""},{"location":"docs/reference/dstack.yml/fleet/#compute_capability","title":"compute_capability - (Optional) The minimum compute capability of the GPU (e.g., 7.5).","text":""},{"location":"docs/reference/dstack.yml/fleet/#resources-disk","title":"resouces.disk","text":""},{"location":"docs/reference/dstack.yml/fleet/#size","title":"size - The disk size. Can be a string (e.g., 100GB or 100GB..) or an object.","text":""},{"location":"docs/reference/dstack.yml/fleet/#retry","title":"retry","text":""},{"location":"docs/reference/dstack.yml/fleet/#on_events","title":"on_events - The list of events that should be handled with retry. Supported events are no-capacity, interruption, and error.","text":""},{"location":"docs/reference/dstack.yml/fleet/#duration","title":"duration - (Optional) The maximum period of retrying the run, e.g., 4h or 1d.","text":""},{"location":"docs/reference/dstack.yml/gateway/","title":"gateway","text":"The gateway configuration type allows creating and updating gateways.
Configuration files must have a name ending with .dstack.yml (e.g., .dstack.yml or gateway.dstack.yml are both acceptable) and can be located in the project's root directory or any nested folder. Any configuration can be applied via dstack apply.
type: gateway\nname: example-gateway\n\nbackend: aws\nregion: eu-west-1\ndomain: example.com\nname - (Optional) The gateway name.","text":""},{"location":"docs/reference/dstack.yml/gateway/#default","title":"default - (Optional) Make the gateway default.","text":""},{"location":"docs/reference/dstack.yml/gateway/#backend","title":"backend - The gateway backend.","text":""},{"location":"docs/reference/dstack.yml/gateway/#region","title":"region - The gateway region.","text":""},{"location":"docs/reference/dstack.yml/gateway/#domain","title":"domain - (Optional) The gateway domain, e.g. example.com.","text":""},{"location":"docs/reference/dstack.yml/gateway/#public_ip","title":"public_ip - (Optional) Allocate public IP for the gateway. Defaults to True.","text":""},{"location":"docs/reference/dstack.yml/gateway/#_certificate","title":"certificate - (Optional) The SSL certificate configuration. Defaults to type: lets-encrypt.","text":""},{"location":"docs/reference/dstack.yml/gateway/#certificatetypelets-encrypt","title":"certificate[type=lets-encrypt]","text":""},{"location":"docs/reference/dstack.yml/gateway/#type","title":"type - Automatic certificates by Let's Encrypt. Must be lets-encrypt.","text":""},{"location":"docs/reference/dstack.yml/gateway/#certificatetypeacm","title":"certificate[type=acm]","text":""},{"location":"docs/reference/dstack.yml/gateway/#type","title":"type - Certificates by AWS Certificate Manager (ACM). Must be acm.","text":""},{"location":"docs/reference/dstack.yml/gateway/#arn","title":"arn - The ARN of the wildcard ACM certificate for the domain.","text":""},{"location":"docs/reference/dstack.yml/service/","title":"service","text":"The service configuration type allows running services.
Configuration files must have a name ending with .dstack.yml (e.g., .dstack.yml or serve.dstack.yml are both acceptable) and can be located in the project's root directory or any nested folder. Any configuration can be run via dstack run . -f PATH.
If you don't specify image, dstack uses the default Docker image pre-configured with python, pip, conda (Miniforge), and essential CUDA drivers. The python property determines which default Docker image is used.
type: service\n\npython: \"3.11\"\n\ncommands:\n - python3 -m http.server\n\nport: 8000\nnvcc
Note that the default Docker image doesn't bundle nvcc, which is required for building custom CUDA kernels. To install it, use conda install cuda.
type: service\n\nimage: dstackai/base:py3.11-0.4-cuda-12.1\n\ncommands:\n - python3 -m http.server\n\nport: 8000\nUse the registry_auth property to provide credentials for a private Docker registry.
type: service\n\nimage: dstackai/base:py3.11-0.4-cuda-12.1\n\ncommands:\n - python3 -m http.server\nregistry_auth:\n username: peterschmidt85\n password: ghp_e49HcZ9oYwBzUbcSk2080gXZOU2hiT9AeSR5\n\nport: 8000\nBy default, if you run a service, its endpoint is accessible at https://<run name>.<gateway domain>.
If you run a model, you can optionally configure the mapping to make it accessible via the OpenAI-compatible interface.
type: service\n\npython: \"3.11\"\n\nenv:\n - MODEL=NousResearch/Llama-2-7b-chat-hf\ncommands:\n - pip install vllm\n - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000\nport: 8000\n\nresources:\n gpu: 24GB\n\n# Enable the OpenAI-compatible endpoint\nmodel:\n format: openai\n type: chat\n name: NousResearch/Llama-2-7b-chat-hf\nIn this case, with such a configuration, once the service is up, you'll be able to access the model at https://gateway.<gateway domain> via the OpenAI-compatible interface.
The format supports only tgi (Text Generation Inference) and openai (if you are using Text Generation Inference or vLLM with OpenAI-compatible mode).
By default, dstack loads the chat template from the model's repository. If it is not present there, manual configuration is required.
type: service\n\nimage: ghcr.io/huggingface/text-generation-inference:latest\nenv:\n - MODEL_ID=TheBloke/Llama-2-13B-chat-GPTQ\ncommands:\n - text-generation-launcher --port 8000 --trust-remote-code --quantize gptq\nport: 8000\n\nresources:\n gpu: 80GB\n\n# Enable the OpenAI-compatible endpoint\nmodel:\n type: chat\n name: TheBloke/Llama-2-13B-chat-GPTQ\n format: tgi\n chat_template: \"{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% set system_message = false %}{% endif %}{% for message in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if loop.index0 == 0 and system_message != false %}{% set content = '<<SYS>>\\\\n' + system_message + '\\\\n<</SYS>>\\\\n\\\\n' + message['content'] %}{% else %}{% set content = message['content'] %}{% endif %}{% if message['role'] == 'user' %}{{ '<s>[INST] ' + content.strip() + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ ' ' + content.strip() + ' </s>' }}{% endif %}{% endfor %}\"\n eos_token: \"</s>\"\nPlease note that model mapping is an experimental feature with the following limitations:
chat_template uses bos_token. As a workaround, replace bos_token inside chat_template with the token content itself.eos_token is defined in the model repository as a dictionary. As a workaround, set eos_token manually, as shown in the example above (see Chat template).If you encounter any other issues, please make sure to file a GitHub issue.
"},{"location":"docs/reference/dstack.yml/service/#auto-scaling","title":"Auto-scaling","text":"By default, dstack runs a single replica of the service. You can configure the number of replicas as well as the auto-scaling rules.
type: service\n\npython: \"3.11\"\n\nenv:\n - MODEL=NousResearch/Llama-2-7b-chat-hf\ncommands:\n - pip install vllm\n - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000\nport: 8000\n\nresources:\n gpu: 24GB\n\n# Enable the OpenAI-compatible endpoint\nmodel:\n format: openai\n type: chat\n name: NousResearch/Llama-2-7b-chat-hf\n\nreplicas: 1..4\nscaling:\n metric: rps\n target: 10\nThe replicas property can be a number or a range.
The metric property of scaling only supports the rps metric (requests per second). In this case dstack adjusts the number of replicas (scales up or down) automatically based on the load.
Setting the minimum number of replicas to 0 allows the service to scale down to zero when there are no requests.
If you specify memory size, you can either specify an explicit size (e.g. 24GB) or a range (e.g. 24GB.., or 24GB..80GB, or ..80GB).
type: service\n\npython: \"3.11\"\ncommands:\n - pip install vllm\n - python -m vllm.entrypoints.openai.api_server\n --model mistralai/Mixtral-8X7B-Instruct-v0.1\n --host 0.0.0.0\n --tensor-parallel-size 2 # Match the number of GPUs\nport: 8000\n\nresources:\n # 2 GPUs of 80GB\n gpu: 80GB:2\n\n disk: 200GB\n\n# Enable the OpenAI-compatible endpoint\nmodel:\n type: chat\n name: TheBloke/Mixtral-8x7B-Instruct-v0.1-GPTQ\n format: openai\nThe gpu property allows specifying not only memory size but also GPU names and their quantity. Examples: A100 (one A100), A10G,A100 (either A10G or A100), A100:80GB (one A100 of 80GB), A100:2 (two A100), 24GB..40GB:2 (two GPUs between 24GB and 40GB), A100:40GB:2 (two A100 GPUs of 40GB).
If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure shm_size, e.g. set it to 16GB.
By default, the service endpoint requires the Authorization header with \"Bearer <dstack token>\". Authorization can be disabled by setting auth to false.
type: service\n\npython: \"3.11\"\n\ncommands:\n - python3 -m http.server\n\nport: 8000\n\nauth: false\ntype: service\n\npython: \"3.11\"\n\nenv:\n - HUGGING_FACE_HUB_TOKEN\n - MODEL=NousResearch/Llama-2-7b-chat-hf\ncommands:\n - pip install vllm\n - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000\nport: 8000\n\nresources:\n gpu: 24GB\nIf you don't assign a value to an environment variable (see HUGGING_FACE_HUB_TOKEN above), dstack will require the value to be passed via the CLI or set in the current process.
For instance, you can define environment variables in a .env file and utilize tools like direnv.
The following environment variables are available in any run and are passed by dstack by default:
DSTACK_RUN_NAME The name of the run DSTACK_REPO_ID The ID of the repo DSTACK_GPUS_NUM The total number of GPUs in the run"},{"location":"docs/reference/dstack.yml/service/#spot-policy","title":"Spot policy","text":"You can choose whether to use spot instances, on-demand instances, or any available type.
type: service\n\ncommands:\n - python3 -m http.server\n\nport: 8000\n\nspot_policy: auto\nThe spot_policy accepts spot, on-demand, and auto. The default for services is auto.
By default, dstack provisions instances in all configured backends. However, you can specify the list of backends:
type: service\n\ncommands:\n - python3 -m http.server\n\nport: 8000\n\nbackends: [aws, gcp]\nBy default, dstack uses all configured regions. However, you can specify the list of regions:
type: service\n\ncommands:\n - python3 -m http.server\n\nport: 8000\n\nregions: [eu-west-1, eu-west-2]\nVolumes allow you to persist data between runs. To attach a volume, simply specify its name using the volumes property and specify where to mount its contents:
type: service\n\ncommands:\n - python3 -m http.server\n\nport: 8000\n\nvolumes:\n - name: my-new-volume\n path: /volume_data\nOnce you run this configuration, the contents of the volume will be attached to /volume_data inside the service, and its contents will persist across runs.
The service configuration type supports many other options. See below.
port - The port, that application listens on or the mapping.","text":""},{"location":"docs/reference/dstack.yml/service/#model","title":"model - (Optional) Mapping of the model for the OpenAI-compatible endpoint.","text":""},{"location":"docs/reference/dstack.yml/service/#https","title":"https - (Optional) Enable HTTPS. Defaults to True.","text":""},{"location":"docs/reference/dstack.yml/service/#auth","title":"auth - (Optional) Enable the authorization. Defaults to True.","text":""},{"location":"docs/reference/dstack.yml/service/#replicas","title":"replicas - (Optional) The number of replicas. Can be a number (e.g. 2) or a range (0..4 or 1..8). If it's a range, the scaling property is required. Defaults to 1.","text":""},{"location":"docs/reference/dstack.yml/service/#_scaling","title":"scaling - (Optional) The auto-scaling rules. Required if replicas is set to a range.","text":""},{"location":"docs/reference/dstack.yml/service/#name","title":"name - (Optional) The run name.","text":""},{"location":"docs/reference/dstack.yml/service/#image","title":"image - (Optional) The name of the Docker image to run.","text":""},{"location":"docs/reference/dstack.yml/service/#entrypoint","title":"entrypoint - (Optional) The Docker entrypoint.","text":""},{"location":"docs/reference/dstack.yml/service/#working_dir","title":"working_dir - (Optional) The path to the working directory inside the container. It's specified relative to the repository directory (/workflow) and should be inside it. Defaults to \".\" .","text":""},{"location":"docs/reference/dstack.yml/service/#home_dir","title":"home_dir - (Optional) The absolute path to the home directory inside the container. Defaults to /root. Defaults to /root.","text":""},{"location":"docs/reference/dstack.yml/service/#_registry_auth","title":"registry_auth - (Optional) Credentials for pulling a private Docker image.","text":""},{"location":"docs/reference/dstack.yml/service/#python","title":"python - (Optional) The major version of Python. Mutually exclusive with image.","text":""},{"location":"docs/reference/dstack.yml/service/#env","title":"env - (Optional) The mapping or the list of environment variables.","text":""},{"location":"docs/reference/dstack.yml/service/#setup","title":"setup - (Optional) The bash commands to run on the boot.","text":""},{"location":"docs/reference/dstack.yml/service/#_resources","title":"resources - (Optional) The resources requirements to run the configuration.","text":""},{"location":"docs/reference/dstack.yml/service/#_volumes","title":"volumes - (Optional) The volumes mount points.","text":""},{"location":"docs/reference/dstack.yml/service/#commands","title":"commands - (Optional) The bash commands to run.","text":""},{"location":"docs/reference/dstack.yml/service/#backends","title":"backends - (Optional) The backends to consider for provisioning (e.g., [aws, gcp]).","text":""},{"location":"docs/reference/dstack.yml/service/#regions","title":"regions - (Optional) The regions to consider for provisioning (e.g., [eu-west-1, us-west4, westeurope]).","text":""},{"location":"docs/reference/dstack.yml/service/#instance_types","title":"instance_types - (Optional) The cloud-specific instance types to consider for provisioning (e.g., [p3.8xlarge, n1-standard-4]).","text":""},{"location":"docs/reference/dstack.yml/service/#spot_policy","title":"spot_policy - (Optional) The policy for provisioning spot or on-demand instances: spot, on-demand, or auto.","text":""},{"location":"docs/reference/dstack.yml/service/#_retry","title":"retry - (Optional) The policy for resubmitting the run. Defaults to false.","text":""},{"location":"docs/reference/dstack.yml/service/#_retry_policy","title":"retry_policy - (Optional) The policy for resubmitting the run. Deprecated in favor of retry.","text":""},{"location":"docs/reference/dstack.yml/service/#max_duration","title":"max_duration - (Optional) The maximum duration of a run (e.g., 2h, 1d, etc). After it elapses, the run is forced to stop. Defaults to off.","text":""},{"location":"docs/reference/dstack.yml/service/#max_price","title":"max_price - (Optional) The maximum instance price per hour, in dollars.","text":""},{"location":"docs/reference/dstack.yml/service/#pool_name","title":"pool_name - (Optional) The name of the pool. If not set, dstack will use the default name.","text":""},{"location":"docs/reference/dstack.yml/service/#instance_name","title":"instance_name - (Optional) The name of the instance.","text":""},{"location":"docs/reference/dstack.yml/service/#creation_policy","title":"creation_policy - (Optional) The policy for using instances from the pool. Defaults to reuse-or-create.","text":""},{"location":"docs/reference/dstack.yml/service/#termination_policy","title":"termination_policy - (Optional) The policy for instance termination. Defaults to destroy-after-idle.","text":""},{"location":"docs/reference/dstack.yml/service/#termination_idle_time","title":"termination_idle_time - (Optional) Time to wait before destroying the idle instance. Defaults to 5m for dstack run and to 3d for dstack pool add.","text":""},{"location":"docs/reference/dstack.yml/service/#model_1","title":"model","text":""},{"location":"docs/reference/dstack.yml/service/#type","title":"type - The type of the model.","text":""},{"location":"docs/reference/dstack.yml/service/#name","title":"name - The name of the model.","text":""},{"location":"docs/reference/dstack.yml/service/#format","title":"format - The serving format. Supported values include openai and tgi.","text":""},{"location":"docs/reference/dstack.yml/service/#scaling","title":"scaling","text":""},{"location":"docs/reference/dstack.yml/service/#metric","title":"metric - The target metric to track. Currently, the only supported value is rps (meaning requests per second).","text":""},{"location":"docs/reference/dstack.yml/service/#target","title":"target - The target value of the metric. The number of replicas is calculated based on this number and automatically adjusts (scales up or down) as this metric changes.","text":""},{"location":"docs/reference/dstack.yml/service/#scale_up_delay","title":"scale_up_delay - (Optional) The delay in seconds before scaling up. Defaults to 300.","text":""},{"location":"docs/reference/dstack.yml/service/#scale_down_delay","title":"scale_down_delay - (Optional) The delay in seconds before scaling down. Defaults to 600.","text":""},{"location":"docs/reference/dstack.yml/service/#resources","title":"resources","text":""},{"location":"docs/reference/dstack.yml/service/#cpu","title":"cpu - (Optional) The number of CPU cores. Defaults to 2...","text":""},{"location":"docs/reference/dstack.yml/service/#memory","title":"memory - (Optional) The RAM size (e.g., 8GB). Defaults to 8GB...","text":""},{"location":"docs/reference/dstack.yml/service/#shm_size","title":"shm_size - (Optional) The size of shared memory (e.g., 8GB). If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure this.","text":""},{"location":"docs/reference/dstack.yml/service/#_gpu","title":"gpu - (Optional) The GPU requirements. Can be set to a number, a string (e.g. A100, 80GB:2, etc.), or an object.","text":""},{"location":"docs/reference/dstack.yml/service/#_disk","title":"disk - (Optional) The disk resources.","text":""},{"location":"docs/reference/dstack.yml/service/#resources-gpu","title":"resouces.gpu","text":""},{"location":"docs/reference/dstack.yml/service/#name","title":"name - (Optional) The GPU name or list of names.","text":""},{"location":"docs/reference/dstack.yml/service/#count","title":"count - (Optional) The number of GPUs. Defaults to 1.","text":""},{"location":"docs/reference/dstack.yml/service/#memory","title":"memory - (Optional) The RAM size (e.g., 16GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).","text":""},{"location":"docs/reference/dstack.yml/service/#total_memory","title":"total_memory - (Optional) The total RAM size (e.g., 32GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).","text":""},{"location":"docs/reference/dstack.yml/service/#compute_capability","title":"compute_capability - (Optional) The minimum compute capability of the GPU (e.g., 7.5).","text":""},{"location":"docs/reference/dstack.yml/service/#resources-disk","title":"resouces.disk","text":""},{"location":"docs/reference/dstack.yml/service/#size","title":"size - The disk size. Can be a string (e.g., 100GB or 100GB..) or an object.","text":""},{"location":"docs/reference/dstack.yml/service/#registry_auth","title":"registry_auth","text":""},{"location":"docs/reference/dstack.yml/service/#username","title":"username - The username.","text":""},{"location":"docs/reference/dstack.yml/service/#password","title":"password - The password or access token.","text":""},{"location":"docs/reference/dstack.yml/service/#volumes_1","title":"volumes","text":""},{"location":"docs/reference/dstack.yml/service/#name","title":"name - The name of the volume to mount.","text":""},{"location":"docs/reference/dstack.yml/service/#path","title":"path - The container path to mount the volume at.","text":""},{"location":"docs/reference/dstack.yml/task/","title":"task","text":"The task configuration type allows running tasks.
Configuration files must have a name ending with .dstack.yml (e.g., .dstack.yml or serve.dstack.yml are both acceptable) and can be located in the project's root directory or any nested folder. Any configuration can be run via dstack run.
If you don't specify image, dstack uses the default Docker image pre-configured with python, pip, conda (Miniforge), and essential CUDA drivers. The python property determines which default Docker image is used.
type: task\n\npython: \"3.11\"\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py\nnvcc
Note that the default Docker image doesn't bundle nvcc, which is required for building custom CUDA kernels. To install it, use conda install cuda.
A task can configure ports. In this case, if the task is running an application on a port, dstack run will securely allow you to access this port from your local machine through port forwarding.
type: task\n\npython: \"3.11\"\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - tensorboard --logdir results/runs &\n - python fine-tuning/qlora/train.py\n\nports:\n - 6000\nWhen running it, dstack run forwards 6000 port to localhost:6000, enabling secure access.
type: dev-environment\n\nimage: dstackai/base:py3.11-0.4-cuda-12.1\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py\nUse the registry_auth property to provide credentials for a private Docker registry.
type: dev-environment\n\nimage: dstackai/base:py3.11-0.4-cuda-12.1\nregistry_auth:\n username: peterschmidt85\n password: ghp_e49HcZ9oYwBzUbcSk2080gXZOU2hiT9AeSR5\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py\nIf you specify memory size, you can either specify an explicit size (e.g. 24GB) or a range (e.g. 24GB.., or 24GB..80GB, or ..80GB).
type: task\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py\n\nresources:\n # 200GB or more RAM\n memory: 200GB..\n\n # 4 GPUs from 40GB to 80GB\n gpu: 40GB..80GB:4\n\n # Shared memory\n shm_size: 16GB\n\n disk: 500GB\nThe gpu property allows specifying not only memory size but also GPU names and their quantity. Examples: A100 (one A100), A10G,A100 (either A10G or A100), A100:80GB (one A100 of 80GB), A100:2 (two A100), 24GB..40GB:2 (two GPUs between 24GB and 40GB), A100:40GB:2 (two A100 GPUs of 40GB).
To use TPUs, specify its architecture prefixed by tpu- via the gpu property.
type: task\n\npython: \"3.11\"\n\ncommands:\n - pip install torch~=2.3.0 torch_xla[tpu]~=2.3.0 torchvision -f https://storage.googleapis.com/libtpu-releases/index.html\n - git clone --recursive https://github.com/pytorch/xla.git\n - python3 xla/test/test_train_mp_imagenet.py --fake_data --model=resnet50 --num_epochs=1\n\nresources:\n gpu: tpu-v2-8\nCurrently, only 8 TPU cores can be specified, supporting single host workloads. Multi-host support is coming soon.
Shared memoryIf you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure shm_size, e.g. set it to 16GB.
type: task\n\npython: \"3.11\"\n\nenv:\n - HUGGING_FACE_HUB_TOKEN\n - HF_HUB_ENABLE_HF_TRANSFER=1\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py\nIf you don't assign a value to an environment variable (see HUGGING_FACE_HUB_TOKEN above), dstack will require the value to be passed via the CLI or set in the current process.
For instance, you can define environment variables in a .env file and utilize tools like direnv.
The following environment variables are available in any run and are passed by dstack by default:
DSTACK_RUN_NAME The name of the run DSTACK_REPO_ID The ID of the repo DSTACK_GPUS_NUM The total number of GPUs in the run DSTACK_NODES_NUM The number of nodes in the run DSTACK_NODE_RANK The rank of the node DSTACK_MASTER_NODE_IP The internal IP address the master node"},{"location":"docs/reference/dstack.yml/task/#_nodes","title":"Distributed tasks","text":"By default, the task runs on a single node. However, you can run it on a cluster of nodes.
type: task\n\n# The size of the cluster\nnodes: 2\n\npython: \"3.11\"\nenv:\n - HF_HUB_ENABLE_HF_TRANSFER=1\ncommands:\n - pip install -r requirements.txt\n - torchrun\n --nproc_per_node=$DSTACK_GPUS_PER_NODE\n --node_rank=$DSTACK_NODE_RANK\n --nnodes=$DSTACK_NODES_NUM\n --master_addr=$DSTACK_MASTER_NODE_IP\n --master_port=8008 resnet_ddp.py\n --num_epochs 20\n\nresources:\n gpu: 24GB\nIf you run the task, dstack first provisions the master node and then runs the other nodes of the cluster. All nodes are provisioned in the same region.
dstack is easy to use with accelerate, torchrun, and other distributed frameworks. All you need to do is pass the corresponding environment variables such as DSTACK_GPUS_PER_NODE, DSTACK_NODE_RANK, DSTACK_NODES_NUM, DSTACK_MASTER_NODE_IP, and DSTACK_GPUS_NUM (see System environment variables).
Running on multiple nodes is supported only with aws, gcp, azure, oci, and instances added via dstack pool add-ssh.
You can parameterize tasks with user arguments using ${{ run.args }} in the configuration.
type: task\n\npython: \"3.11\"\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py ${{ run.args }}\nNow, you can pass your arguments to the dstack run command:
$ dstack run . -f train.dstack.yml --train_batch_size=1 --num_train_epochs=100\nHere's an example of using ports to run web apps with tasks.
type: task\n\npython: \"3.11\"\n\ncommands:\n - pip3 install streamlit\n - streamlit hello\n\nports: \n - 8501\nYou can choose whether to use spot instances, on-demand instances, or any available type.
type: task\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py\n\nspot_policy: auto\nThe spot_policy accepts spot, on-demand, and auto. The default for tasks is auto.
By default, dstack provisions instances in all configured backends. However, you can specify the list of backends:
type: task\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py\n\nbackends: [aws, gcp]\nBy default, dstack uses all configured regions. However, you can specify the list of regions:
type: task\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py\n\nregions: [eu-west-1, eu-west-2]\nVolumes allow you to persist data between runs. To attach a volume, simply specify its name using the volumes property and specify where to mount its contents:
type: task\n\npython: \"3.11\"\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py\n\nvolumes:\n - name: my-new-volume\n path: /volume_data\nOnce you run this configuration, the contents of the volume will be attached to /volume_data inside the task, and its contents will persist across runs.
Limitations
When you're running a dev environment, task, or service with dstack, it automatically mounts the project folder contents to /workflow (and sets that as the current working directory). Right now, dstack doesn't allow you to attach volumes to /workflow or any of its subdirectories.
The task configuration type supports many other options. See below.
nodes - (Optional) Number of nodes. Defaults to 1.","text":""},{"location":"docs/reference/dstack.yml/task/#name","title":"name - (Optional) The run name.","text":""},{"location":"docs/reference/dstack.yml/task/#image","title":"image - (Optional) The name of the Docker image to run.","text":""},{"location":"docs/reference/dstack.yml/task/#entrypoint","title":"entrypoint - (Optional) The Docker entrypoint.","text":""},{"location":"docs/reference/dstack.yml/task/#working_dir","title":"working_dir - (Optional) The path to the working directory inside the container. It's specified relative to the repository directory (/workflow) and should be inside it. Defaults to \".\" .","text":""},{"location":"docs/reference/dstack.yml/task/#home_dir","title":"home_dir - (Optional) The absolute path to the home directory inside the container. Defaults to /root. Defaults to /root.","text":""},{"location":"docs/reference/dstack.yml/task/#_registry_auth","title":"registry_auth - (Optional) Credentials for pulling a private Docker image.","text":""},{"location":"docs/reference/dstack.yml/task/#python","title":"python - (Optional) The major version of Python. Mutually exclusive with image.","text":""},{"location":"docs/reference/dstack.yml/task/#env","title":"env - (Optional) The mapping or the list of environment variables.","text":""},{"location":"docs/reference/dstack.yml/task/#setup","title":"setup - (Optional) The bash commands to run on the boot.","text":""},{"location":"docs/reference/dstack.yml/task/#_resources","title":"resources - (Optional) The resources requirements to run the configuration.","text":""},{"location":"docs/reference/dstack.yml/task/#_volumes","title":"volumes - (Optional) The volumes mount points.","text":""},{"location":"docs/reference/dstack.yml/task/#ports","title":"ports - (Optional) Port numbers/mapping to expose.","text":""},{"location":"docs/reference/dstack.yml/task/#commands","title":"commands - (Optional) The bash commands to run.","text":""},{"location":"docs/reference/dstack.yml/task/#backends","title":"backends - (Optional) The backends to consider for provisioning (e.g., [aws, gcp]).","text":""},{"location":"docs/reference/dstack.yml/task/#regions","title":"regions - (Optional) The regions to consider for provisioning (e.g., [eu-west-1, us-west4, westeurope]).","text":""},{"location":"docs/reference/dstack.yml/task/#instance_types","title":"instance_types - (Optional) The cloud-specific instance types to consider for provisioning (e.g., [p3.8xlarge, n1-standard-4]).","text":""},{"location":"docs/reference/dstack.yml/task/#spot_policy","title":"spot_policy - (Optional) The policy for provisioning spot or on-demand instances: spot, on-demand, or auto.","text":""},{"location":"docs/reference/dstack.yml/task/#_retry","title":"retry - (Optional) The policy for resubmitting the run. Defaults to false.","text":""},{"location":"docs/reference/dstack.yml/task/#_retry_policy","title":"retry_policy - (Optional) The policy for resubmitting the run. Deprecated in favor of retry.","text":""},{"location":"docs/reference/dstack.yml/task/#max_duration","title":"max_duration - (Optional) The maximum duration of a run (e.g., 2h, 1d, etc). After it elapses, the run is forced to stop. Defaults to off.","text":""},{"location":"docs/reference/dstack.yml/task/#max_price","title":"max_price - (Optional) The maximum instance price per hour, in dollars.","text":""},{"location":"docs/reference/dstack.yml/task/#pool_name","title":"pool_name - (Optional) The name of the pool. If not set, dstack will use the default name.","text":""},{"location":"docs/reference/dstack.yml/task/#instance_name","title":"instance_name - (Optional) The name of the instance.","text":""},{"location":"docs/reference/dstack.yml/task/#creation_policy","title":"creation_policy - (Optional) The policy for using instances from the pool. Defaults to reuse-or-create.","text":""},{"location":"docs/reference/dstack.yml/task/#termination_policy","title":"termination_policy - (Optional) The policy for instance termination. Defaults to destroy-after-idle.","text":""},{"location":"docs/reference/dstack.yml/task/#termination_idle_time","title":"termination_idle_time - (Optional) Time to wait before destroying the idle instance. Defaults to 5m for dstack run and to 3d for dstack pool add.","text":""},{"location":"docs/reference/dstack.yml/task/#resources","title":"resources","text":""},{"location":"docs/reference/dstack.yml/task/#cpu","title":"cpu - (Optional) The number of CPU cores. Defaults to 2...","text":""},{"location":"docs/reference/dstack.yml/task/#memory","title":"memory - (Optional) The RAM size (e.g., 8GB). Defaults to 8GB...","text":""},{"location":"docs/reference/dstack.yml/task/#shm_size","title":"shm_size - (Optional) The size of shared memory (e.g., 8GB). If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure this.","text":""},{"location":"docs/reference/dstack.yml/task/#_gpu","title":"gpu - (Optional) The GPU requirements. Can be set to a number, a string (e.g. A100, 80GB:2, etc.), or an object.","text":""},{"location":"docs/reference/dstack.yml/task/#_disk","title":"disk - (Optional) The disk resources.","text":""},{"location":"docs/reference/dstack.yml/task/#resources-gpu","title":"resouces.gpu","text":""},{"location":"docs/reference/dstack.yml/task/#name","title":"name - (Optional) The GPU name or list of names.","text":""},{"location":"docs/reference/dstack.yml/task/#count","title":"count - (Optional) The number of GPUs. Defaults to 1.","text":""},{"location":"docs/reference/dstack.yml/task/#memory","title":"memory - (Optional) The RAM size (e.g., 16GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).","text":""},{"location":"docs/reference/dstack.yml/task/#total_memory","title":"total_memory - (Optional) The total RAM size (e.g., 32GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).","text":""},{"location":"docs/reference/dstack.yml/task/#compute_capability","title":"compute_capability - (Optional) The minimum compute capability of the GPU (e.g., 7.5).","text":""},{"location":"docs/reference/dstack.yml/task/#resources-disk","title":"resouces.disk","text":""},{"location":"docs/reference/dstack.yml/task/#size","title":"size - The disk size. Can be a string (e.g., 100GB or 100GB..) or an object.","text":""},{"location":"docs/reference/dstack.yml/task/#registry_auth","title":"registry_auth","text":""},{"location":"docs/reference/dstack.yml/task/#username","title":"username - The username.","text":""},{"location":"docs/reference/dstack.yml/task/#password","title":"password - The password or access token.","text":""},{"location":"docs/reference/dstack.yml/task/#volumesn","title":"volumes[n]","text":""},{"location":"docs/reference/dstack.yml/task/#name","title":"name - The name of the volume to mount.","text":""},{"location":"docs/reference/dstack.yml/task/#path","title":"path - The container path to mount the volume at.","text":""},{"location":"docs/reference/dstack.yml/volume/","title":"volume","text":"The volume configuration type allows creating, registering, and updating volumes.
Configuration files must have a name ending with .dstack.yml (e.g., .dstack.yml or vol.dstack.yml are both acceptable) and can be located in the project's root directory or any nested folder. Any configuration can be applied via dstack apply.
type: volume\nname: my-aws-volume\nbackend: aws\nregion: eu-central-1\nsize: 100GB\ntype: volume\nname: my-external-volume\nbackend: aws\nregion: eu-central-1\nvolume_id: vol1235\nname - (Optional) The volume name.","text":""},{"location":"docs/reference/dstack.yml/volume/#backend","title":"backend - The volume backend.","text":""},{"location":"docs/reference/dstack.yml/volume/#region","title":"region - The volume region.","text":""},{"location":"docs/reference/dstack.yml/volume/#size","title":"size - (Optional) The volume size. Must be specified when creating new volumes.","text":""},{"location":"docs/reference/dstack.yml/volume/#volume_id","title":"volume_id - (Optional) The volume ID. Must be specified when registering external volumes.","text":""},{"location":"docs/reference/server/config.yml/","title":"~/.dstack/server/config.yml","text":"The ~/.dstack/server/config.yml file is used by the dstack server to configure cloud accounts.
The dstack server allows you to configure backends for multiple projects. If you don't need multiple projects, use only the main project.
Each cloud account must be configured under the backends property of the respective project. See the examples below.
There are two ways to configure AWS: using an access key or using the default credentials.
Access keyDefault credentialsCreate an access key by following the this guide . Once you've downloaded the .csv file with your IAM user's Access key ID and Secret access key, proceed to configure the backend.
projects:\n- name: main\n backends:\n - type: aws\n creds:\n type: access_key\n access_key: KKAAUKLIZ5EHKICAOASV\n secret_key: pn158lMqSBJiySwpQ9ubwmI6VUU3/W2fdJdFwfgO\nIf you have default credentials set up (e.g. in ~/.aws/credentials), configure the backend like this:
projects:\n - name: main\n backends:\n - type: aws\n creds:\n type: default\nBy default, dstack uses the default VPC. It's possible to customize it:
projects:\n - name: main\n backends:\n - type: aws\n creds:\n type: default\n\n vpc_name: my-vpc\nprojects:\n - name: main\n backends:\n - type: aws\n creds:\n type: default\n\n default_vpcs: true\n vpc_ids:\n us-east-1: vpc-0a2b3c4d5e6f7g8h\n us-east-2: vpc-9i8h7g6f5e4d3c2b\n us-west-1: vpc-4d3c2b1a0f9e8d7\nFor the regions without configured vpc_ids, enable default VPCs by setting default_vpcs to true.
The following AWS policy permissions are sufficient for dstack to work:
{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"ec2:AttachVolume\",\n \"ec2:AuthorizeSecurityGroupEgress\",\n \"ec2:AuthorizeSecurityGroupIngress\",\n \"ec2:CancelSpotInstanceRequests\",\n \"ec2:CreateSecurityGroup\",\n \"ec2:CreateTags\",\n \"ec2:CreateVolume\",\n \"ec2:DeleteVolume\",\n \"ec2:DescribeAvailabilityZones\",\n \"ec2:DescribeImages\",\n \"ec2:DescribeInstances\",\n \"ec2:DescribeInstanceAttribute\",\n \"ec2:DescribeRouteTables\",\n \"ec2:DescribeSecurityGroups\",\n \"ec2:DescribeSubnets\",\n \"ec2:DescribeVpcs\",\n \"ec2:DescribeVolumes\",\n \"ec2:DetachVolume\",\n \"ec2:RunInstances\",\n \"ec2:TerminateInstances\"\n ],\n \"Resource\": \"*\"\n },\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"servicequotas:ListServiceQuotas\",\n \"servicequotas:GetServiceQuota\"\n ],\n \"Resource\": \"*\"\n },\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"elasticloadbalancing:CreateLoadBalancer\",\n \"elasticloadbalancing:CreateTargetGroup\",\n \"elasticloadbalancing:CreateListener\",\n \"elasticloadbalancing:RegisterTargets\",\n \"elasticloadbalancing:AddTags\",\n \"elasticloadbalancing:DeleteLoadBalancer\",\n \"elasticloadbalancing:DeleteTargetGroup\",\n \"elasticloadbalancing:DeleteListener\",\n \"elasticloadbalancing:DeregisterTargets\"\n ],\n \"Resource\": \"*\"\n },\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"acm:DescribeCertificate\",\n \"acm:ListCertificates\"\n ],\n \"Resource\": \"*\"\n }\n ]\n}\nThe elasticloadbalancing:* and acm:* permissions are only needed for provisioning gateways with ACM (AWS Certificate Manager) certificates.
By default, dstack utilizes public subnets and permits inbound SSH traffic exclusively for any provisioned instances. If you want dstack to use private subnets, set public_ips to false.
projects:\n - name: main\n backends:\n - type: aws\n creds:\n type: default\n\n public_ips: false\nUsing private subnets assumes that both the dstack server and users can access the configured VPC's private subnets (e.g., through VPC peering).
There are two ways to configure Azure: using a client secret or using the default credentials.
Client secretDefault credentialsA client secret can be created using the Azure CLI :
SUBSCRIPTION_ID=...\naz ad sp create-for-rbac\n --name dstack-app \\\n --role $DSTACK_ROLE \\\n --scopes /subscriptions/$SUBSCRIPTION_ID \\\n --query \"{ tenant_id: tenant, client_id: appId, client_secret: password }\"\nOnce you have tenant_id, client_id, and client_secret, go ahead and configure the backend.
projects:\n- name: main\n backends:\n - type: azure\n subscription_id: 06c82ce3-28ff-4285-a146-c5e981a9d808\n tenant_id: f84a7584-88e4-4fd2-8e97-623f0a715ee1\n creds:\n type: client\n client_id: acf3f73a-597b-46b6-98d9-748d75018ed0\n client_secret: 1Kb8Q~o3Q2hdEvrul9yaj5DJDFkuL3RG7lger2VQ\nObtain the subscription_id and tenant_id via the Azure CLI :
az account show --query \"{subscription_id: id, tenant_id: tenantId}\"\nThen proceed to configure the backend:
projects:\n - name: main\n backends:\n - type: azure\n subscription_id: 06c82ce3-28ff-4285-a146-c5e981a9d808\n tenant_id: f84a7584-88e4-4fd2-8e97-623f0a715ee1\n creds:\n type: default\nIf you don't know your subscription_id, run
az account show --query \"{subscription_id: id}\"\nThe following Azure permissions are sufficient for dstack to work:
{\n \"properties\": {\n \"roleName\": \"dstack-role\",\n \"description\": \"Minimal required permissions for using Azure with dstack\",\n \"assignableScopes\": [\n \"/subscriptions/${YOUR_SUBSCRIPTION_ID}\"\n ],\n \"permissions\": [\n {\n \"actions\": [\n \"Microsoft.Authorization/*/read\",\n \"Microsoft.Compute/availabilitySets/*\",\n \"Microsoft.Compute/locations/*\",\n \"Microsoft.Compute/virtualMachines/*\",\n \"Microsoft.Compute/virtualMachineScaleSets/*\",\n \"Microsoft.Compute/cloudServices/*\",\n \"Microsoft.Compute/disks/write\",\n \"Microsoft.Compute/disks/read\",\n \"Microsoft.Compute/disks/delete\",\n \"Microsoft.Network/networkSecurityGroups/*\",\n \"Microsoft.Network/locations/*\",\n \"Microsoft.Network/virtualNetworks/*\",\n \"Microsoft.Network/networkInterfaces/*\",\n \"Microsoft.Network/publicIPAddresses/*\",\n \"Microsoft.Resources/subscriptions/resourceGroups/read\",\n \"Microsoft.Resources/subscriptions/resourceGroups/write\",\n \"Microsoft.Resources/subscriptions/read\"\n ],\n \"notActions\": [],\n \"dataActions\": [],\n \"notDataActions\": []\n }\n ]\n }\n}\nFirst, ensure the required APIs are enabled in your GCP project_id.
PROJECT_ID=...\ngcloud config set project $PROJECT_ID\ngcloud services enable cloudapis.googleapis.com\ngcloud services enable compute.googleapis.com\nThere are two ways to configure GCP: using a service account or using the default credentials.
Service accountDefault credentialsTo create a service account, follow this guide . After setting up the service account create a key for it and download the corresponding JSON file.
Then go ahead and configure the backend by specifying the downloaded file path.
projects:\n- name: main\n backends:\n - type: gcp\n project_id: gcp-project-id\n creds:\n type: service_account\n filename: ~/.dstack/server/gcp-024ed630eab5.json\nEnable GCP application default credentials:
gcloud auth application-default login \nThen configure the backend like this:
projects:\n- name: main\n backends:\n - type: gcp\n project_id: gcp-project-id\n creds:\n type: default\nIf you don't know your GCP project ID, run
gcloud projects list --format=\"json(projectId)\"\nprojects:\n- name: main\n backends:\n - type: gcp\n project_id: gcp-project-id\n creds:\n type: default\n\n vpc_name: my-custom-vpc\nprojects:\n- name: main\n backends:\n - type: gcp\n project_id: gcp-project-id\n creds:\n type: default\n\n vpc_name: my-custom-vpc\n vpc_project_id: another-project-id\nTo use a shared VPC, that VPC has to be configured with two additional firewall rules:
INGRESS traffic on port 22, with the target tag dstack-runner-instanceINGRESS traffic on ports 22, 80, 443, with the target tag dstack-gateway-instanceThe following GCP permissions are sufficient for dstack to work:
compute.disks.create\ncompute.firewalls.create\ncompute.images.useReadOnly\ncompute.instances.create\ncompute.instances.delete\ncompute.instances.get\ncompute.instances.setLabels\ncompute.instances.setMetadata\ncompute.instances.setTags\ncompute.networks.get\ncompute.networks.updatePolicy\ncompute.regions.list\ncompute.subnetworks.list\ncompute.subnetworks.use\ncompute.subnetworks.useExternalIp\ncompute.zoneOperations.get\nIf you plan to use TPUs, additional permissions are required:
tpu.nodes.create\ntpu.nodes.delete\ntpu.nodes.get\ntpu.operations.get\ntpu.operations.list\nAlso, the use of TPUs requires the serviceAccountUser role. For TPU VMs, dstack will use the default service account.
By default, dstack utilizes public subnets and permits inbound SSH traffic exclusively for any provisioned instances. If you want dstack to use private subnets, set public_ips to false.
projects:\n - name: main\n backends:\n - type: gcp\n creds:\n type: default\n\n public_ips: false\nUsing private subnets assumes that both the dstack server and users can access the configured VPC's private subnets (e.g., through VPC peering). Additionally, Cloud NAT must be configured to provide access to external resources for provisioned instances.
There are two ways to configure OCI: using client credentials or using the default credentials.
Client credentialsDefault credentialsLog into the OCI Console , go to My profile, select API keys, and click Add API key.
Once you add a key, you'll see the configuration file. Copy its values to configure the backend as follows:
projects:\n- name: main\n backends:\n - type: oci\n creds:\n type: client\n user: ocid1.user.oc1..g5vlaeqfu47akmaafq665xsgmyaqjktyfxtacfxc4ftjxuca7aohnd2ev66m\n tenancy: ocid1.tenancy.oc1..ajqsftvk4qarcfaak3ha4ycdsaahxmaita5frdwg3tqo2bcokpd3n7oizwai\n region: eu-frankfurt-1\n fingerprint: 77:32:77:00:49:7c:cb:56:84:75:8e:77:96:7d:53:17\n key_file: ~/.oci/private_key.pem\nMake sure to include either the path to your private key via key_file or the contents of the key via key_content.
If you have default credentials set up in ~/.oci/config, configure the backend like this:
projects:\n- name: main\n backends:\n - type: oci\n creds:\n type: default\nThis is an example of a restrictive policy for a group of dstack users:
Allow group <dstack-users> to read compartments in tenancy where target.compartment.name = '<dstack-compartment>'\nAllow group <dstack-users> to read marketplace-community-listings in compartment <dstack-compartment>\nAllow group <dstack-users> to manage app-catalog-listing in compartment <dstack-compartment>\nAllow group <dstack-users> to manage instances in compartment <dstack-compartment>\nAllow group <dstack-users> to manage compute-capacity-reports in compartment <dstack-compartment>\nAllow group <dstack-users> to manage volumes in compartment <dstack-compartment>\nAllow group <dstack-users> to manage volume-attachments in compartment <dstack-compartment>\nAllow group <dstack-users> to manage virtual-network-family in compartment <dstack-compartment>\nTo use this policy, create a compartment for dstack and specify it in ~/.dstack/server/config.yml.
projects:\n- name: main\n backends:\n - type: oci\n creds:\n type: default\n compartment_id: ocid1.compartment.oc1..aaaaaaaa\nLog into your Lambda Cloud account, click API keys in the sidebar, and then click the Generate API key button to create a new API key.
Then, go ahead and configure the backend:
projects:\n- name: main\n backends:\n - type: lambda\n creds:\n type: api_key\n api_key: eersct_yrpiey-naaeedst-tk-_cb6ba38e1128464aea9bcc619e4ba2a5.iijPMi07obgt6TZ87v5qAEj61RVxhd0p\nLog into your TensorDock account, click API in the sidebar, and use the Create an Authorization section to create a new authorization key.
Then, go ahead and configure the backend:
projects:\n - name: main\n backends:\n - type: tensordock\n creds:\n type: api_key\n api_key: 248e621d-9317-7494-dc1557fa5825b-98b\n api_token: FyBI3YbnFEYXdth2xqYRnQI7hiusssBC\nThe tensordock backend supports on-demand instances only. Spot instance support coming soon.
Log into your Vast.ai account, click Account in the sidebar, and copy your API Key.
Then, go ahead and configure the backend:
projects:\n- name: main\n backends:\n - type: vastai\n creds:\n type: api_key\n api_key: d75789f22f1908e0527c78a283b523dd73051c8c7d05456516fc91e9d4efd8c5\nAlso, the vastai backend supports on-demand instances only. Spot instance support coming soon.
Log into your RunPod console, click Settings in the sidebar, expand the API Keys section, and click the button to create a key.
Then proceed to configuring the backend.
projects:\n - name: main\n backends:\n - type: runpod\n creds:\n type: api_key\n api_key: US9XTPDIV8AR42MMINY8TCKRB8S4E7LNRQ6CAUQ9\nLog into your CUDO Compute account, click API keys in the sidebar, and click the Create an API key button.
Ensure you've created a project with CUDO Compute, then proceed to configuring the backend.
projects:\n - name: main\n backends:\n - type: cudo\n project_id: my-cudo-project\n creds:\n type: api_key\n api_key: 7487240a466624b48de22865589\nLog into your DataCrunch account, click Account Settings in the sidebar, find REST API Credentials area and then click the Generate Credentials button.
Then, go ahead and configure the backend:
projects:\n - name: main\n backends:\n - type: datacrunch\n creds:\n type: api_key\n client_id: xfaHBqYEsArqhKWX-e52x3HH7w8T\n client_secret: B5ZU5Qx9Nt8oGMlmMhNI3iglK8bjMhagTbylZy4WzncZe39995f7Vxh8\ndstack supports both self-managed, and managed Kubernetes clusters.
To use GPUs with Kubernetes, the cluster must be installed with the NVIDIA GPU Operator .
To configure a Kubernetes backend, specify the path to the kubeconfig file, and the port that dstack can use for proxying SSH traffic. In case of a self-managed cluster, also specify the IP address of any node in the cluster.
Here's how to configure the backend to use a self-managed cluster.
projects:\n- name: main\n backends:\n - type: kubernetes\n kubeconfig:\n filename: ~/.kube/config\n networking:\n ssh_host: localhost # The external IP address of any node\n ssh_port: 32000 # Any port accessible outside of the cluster\nThe port specified to ssh_port must be accessible outside of the cluster.
If you are using Kind, make sure to make to set up ssh_port via extraPortMappings for proxying SSH traffic:
kind: Cluster\napiVersion: kind.x-k8s.io/v1alpha4\nnodes:\n - role: control-plane\n extraPortMappings:\n - containerPort: 32000 # Must be same as `ssh_port`\n hostPort: 32000 # Must be same as `ssh_port`\nGo ahead and create the cluster like this:
kind create cluster --config examples/misc/kubernetes/kind-config.yml\nHere's how to configure the backend to use a managed cluster (AWS, GCP, Azure).
projects:\n - name: main\n backends:\n - type: kubernetes\n kubeconfig:\n filename: ~/.kube/config\n networking:\n ssh_port: 32000 # Any port accessible outside of the cluster\nThe port specified to ssh_port must be accessible outside of the cluster.
For example, if you are using EKS, make sure to add it via an ingress rule of the corresponding security group:
aws ec2 authorize-security-group-ingress --group-id <cluster-security-group-id> --protocol tcp --port 32000 --cidr 0.0.0.0/0\nprojects - The list of projects.","text":""},{"location":"docs/reference/server/config.yml/#projects","title":"projects[n]","text":""},{"location":"docs/reference/server/config.yml/#name","title":"name - The name of the project.","text":""},{"location":"docs/reference/server/config.yml/#backends","title":"backends - The list of backends.","text":""},{"location":"docs/reference/server/config.yml/#aws","title":"projects[n].backends[type=aws]","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of the backend. Must be aws.","text":""},{"location":"docs/reference/server/config.yml/#regions","title":"regions - (Optional) The list of AWS regions.","text":""},{"location":"docs/reference/server/config.yml/#vpc_name","title":"vpc_name - (Optional) The VPC name. All configured regions must have a VPC with this name.","text":""},{"location":"docs/reference/server/config.yml/#vpc_ids","title":"vpc_ids - (Optional) The mapping from AWS regions to VPC IDs. If default_vpcs: true, omitted regions will use default VPCs.","text":""},{"location":"docs/reference/server/config.yml/#default_vpcs","title":"default_vpcs - (Optional) A flag to enable/disable using default VPCs in regions not configured by vpc_ids. Set to false if default VPCs should never be used. Defaults to true.","text":""},{"location":"docs/reference/server/config.yml/#public_ips","title":"public_ips - (Optional) A flag to enable/disable public IP assigning on instances. Defaults to true.","text":""},{"location":"docs/reference/server/config.yml/#_creds","title":"creds - The credentials.","text":""},{"location":"docs/reference/server/config.yml/#aws-creds","title":"projects[n].backends[type=aws].creds","text":"Access keyDefault"},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be access_key.","text":""},{"location":"docs/reference/server/config.yml/#access_key","title":"access_key - The access key.","text":""},{"location":"docs/reference/server/config.yml/#secret_key","title":"secret_key - The secret key.","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be default.","text":""},{"location":"docs/reference/server/config.yml/#azure","title":"projects[n].backends[type=azure]","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of the backend. Must be azure.","text":""},{"location":"docs/reference/server/config.yml/#tenant_id","title":"tenant_id - The tenant ID.","text":""},{"location":"docs/reference/server/config.yml/#subscription_id","title":"subscription_id - The subscription ID.","text":""},{"location":"docs/reference/server/config.yml/#_creds","title":"creds - The credentials.","text":""},{"location":"docs/reference/server/config.yml/#azure-creds","title":"projects[n].backends[type=azure].creds","text":"ClientDefault"},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be client.","text":""},{"location":"docs/reference/server/config.yml/#client_id","title":"client_id - The client ID.","text":""},{"location":"docs/reference/server/config.yml/#client_secret","title":"client_secret - The client secret.","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be default.","text":""},{"location":"docs/reference/server/config.yml/#datacrunch","title":"projects[n].backends[type=datacrunch]","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of backend. Must be datacrunch.","text":""},{"location":"docs/reference/server/config.yml/#_creds","title":"creds - The credentials.","text":""},{"location":"docs/reference/server/config.yml/#datacrunch-creds","title":"projects[n].backends[type=datacrunch].creds","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be api_key.","text":""},{"location":"docs/reference/server/config.yml/#client_id","title":"client_id - The client ID.","text":""},{"location":"docs/reference/server/config.yml/#client_secret","title":"client_secret - The client secret.","text":""},{"location":"docs/reference/server/config.yml/#gcp","title":"projects[n].backends[type=gcp]","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of backend. Must be gcp.","text":""},{"location":"docs/reference/server/config.yml/#project_id","title":"project_id - The project ID.","text":""},{"location":"docs/reference/server/config.yml/#vpc_name","title":"vpc_name - (Optional) The VPC name.","text":""},{"location":"docs/reference/server/config.yml/#vpc_project_id","title":"vpc_project_id - (Optional) The shared VPC hosted project ID. Required for shared VPC only.","text":""},{"location":"docs/reference/server/config.yml/#public_ips","title":"public_ips - (Optional) A flag to enable/disable public IP assigning on instances. Defaults to true.","text":""},{"location":"docs/reference/server/config.yml/#_creds","title":"creds - The credentials.","text":""},{"location":"docs/reference/server/config.yml/#gcp-creds","title":"projects[n].backends[type=gcp].creds","text":"Service accountDefault"},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be service_account.","text":""},{"location":"docs/reference/server/config.yml/#filename","title":"filename - The path to the service account file.","text":""},{"location":"docs/reference/server/config.yml/#data","title":"data - (Optional) The contents of the service account file.","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be default.","text":""},{"location":"docs/reference/server/config.yml/#lambda","title":"projects[n].backends[type=lambda]","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of backend. Must be lambda.","text":""},{"location":"docs/reference/server/config.yml/#_creds","title":"creds - The credentials.","text":""},{"location":"docs/reference/server/config.yml/#lambda-creds","title":"projects[n].backends[type=lambda].creds","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be api_key.","text":""},{"location":"docs/reference/server/config.yml/#api_key","title":"api_key - The API key.","text":""},{"location":"docs/reference/server/config.yml/#oci","title":"projects[n].backends[type=oci]","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of backend. Must be oci.","text":""},{"location":"docs/reference/server/config.yml/#_creds","title":"creds - The credentials.","text":""},{"location":"docs/reference/server/config.yml/#regions","title":"regions - (Optional) List of region names for running dstack jobs. Omit to use all regions.","text":""},{"location":"docs/reference/server/config.yml/#compartment_id","title":"compartment_id - (Optional) Compartment where dstack will create all resources. Omit to instruct dstack to create a new compartment.","text":""},{"location":"docs/reference/server/config.yml/#oci-creds","title":"projects[n].backends[type=oci].creds","text":"ClientDefault"},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be client.","text":""},{"location":"docs/reference/server/config.yml/#user","title":"user - User OCID.","text":""},{"location":"docs/reference/server/config.yml/#tenancy","title":"tenancy - Tenancy OCID.","text":""},{"location":"docs/reference/server/config.yml/#key_file","title":"key_file - (Optional) Path to the user's private PEM key. Either this or key_content should be set.","text":""},{"location":"docs/reference/server/config.yml/#key_content","title":"key_content - (Optional) Content of the user's private PEM key. Either this or key_file should be set.","text":""},{"location":"docs/reference/server/config.yml/#pass_phrase","title":"pass_phrase - (Optional) Passphrase for the private PEM key if it is encrypted.","text":""},{"location":"docs/reference/server/config.yml/#fingerprint","title":"fingerprint - User's public key fingerprint.","text":""},{"location":"docs/reference/server/config.yml/#region","title":"region - Name or key of any region the tenancy is subscribed to.","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be default.","text":""},{"location":"docs/reference/server/config.yml/#file","title":"file - (Optional) Path to the OCI CLI-compatible config file. Defaults to ~/.oci/config.","text":""},{"location":"docs/reference/server/config.yml/#profile","title":"profile - (Optional) Profile to load from the config file. Defaults to DEFAULT.","text":""},{"location":"docs/reference/server/config.yml/#tensordock","title":"projects[n].backends[type=tensordock]","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of backend. Must be tensordock.","text":""},{"location":"docs/reference/server/config.yml/#_creds","title":"creds - The credentials.","text":""},{"location":"docs/reference/server/config.yml/#tensordock-creds","title":"projects[n].backends[type=tensordock].creds","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be api_key.","text":""},{"location":"docs/reference/server/config.yml/#api_key","title":"api_key - The API key.","text":""},{"location":"docs/reference/server/config.yml/#api_token","title":"api_token - The API token.","text":""},{"location":"docs/reference/server/config.yml/#vastai","title":"projects[n].backends[type=vastai]","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of backend. Must be vastai.","text":""},{"location":"docs/reference/server/config.yml/#_creds","title":"creds - The credentials.","text":""},{"location":"docs/reference/server/config.yml/#vastai-creds","title":"projects[n].backends[type=vastai].creds","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be api_key.","text":""},{"location":"docs/reference/server/config.yml/#api_key","title":"api_key - The API key.","text":""},{"location":"docs/reference/server/config.yml/#cudo","title":"projects[n].backends[type=cudo]","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of backend. Must be cudo.","text":""},{"location":"docs/reference/server/config.yml/#project_id","title":"project_id - The project ID.","text":""},{"location":"docs/reference/server/config.yml/#_creds","title":"creds - The credentials.","text":""},{"location":"docs/reference/server/config.yml/#cudo-creds","title":"projects[n].backends[type=cudo].creds","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be api_key.","text":""},{"location":"docs/reference/server/config.yml/#api_key","title":"api_key - The API key.","text":""},{"location":"docs/reference/server/config.yml/#kubernetes","title":"projects[n].backends[type=kubernetes]","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of backend. Must be kubernetes.","text":""},{"location":"docs/reference/server/config.yml/#_kubeconfig","title":"kubeconfig - The kubeconfig configuration.","text":""},{"location":"docs/reference/server/config.yml/#_networking","title":"networking - (Optional) The networking configuration.","text":""},{"location":"docs/reference/server/config.yml/#kubeconfig","title":"projects[n].backends[type=kubernetes].kubeconfig","text":""},{"location":"docs/reference/server/config.yml/#filename","title":"filename - The path to the kubeconfig file.","text":""},{"location":"docs/reference/server/config.yml/#data","title":"data - (Optional) The contents of the kubeconfig file.","text":""},{"location":"docs/reference/server/config.yml/#networking","title":"projects[n].backends[type=kubernetes].networking","text":""},{"location":"docs/reference/server/config.yml/#ssh_host","title":"ssh_host - (Optional) The external IP address of any node.","text":""},{"location":"docs/reference/server/config.yml/#ssh_port","title":"ssh_port - (Optional) Any port accessible outside of the cluster.","text":""},{"location":"blog/archive/2024/","title":"2024","text":""}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"],"fields":{"title":{"boost":1000.0},"text":{"boost":1.0},"tags":{"boost":1000000.0}}},"docs":[{"location":"terms/","title":"Terms of service","text":""},{"location":"terms/#agreement-to-terms","title":"Agreement to terms","text":"We are dstack GmbH (\"Company,\" \"we,\" \"us,\" \"our\"), a company registered in Germany at Franz-Joseph-Stra\u00dfe, 11, Munich, Bayern 80801.
These Legal Terms constitute a legally binding agreement made between you, whether personally or on behalf of an entity (\"you\"), and dstack GmbH, concerning your access to and use of the Services. You agree that by accessing the Services, you have read, understood, and agreed to be bound by all of these Legal Terms. IF YOU DO NOT AGREE WITH ALL OF THESE LEGAL TERMS, THEN YOU ARE EXPRESSLY PROHIBITED FROM USING THE SERVICES AND YOU MUST DISCONTINUE USE IMMEDIATELY.
Supplemental terms and conditions or documents that may be posted on the Services from time to time are hereby expressly incorporated herein by reference. We reserve the right, in our sole discretion, to make changes or modifications to these Legal Terms from time to time. We will alert you about any changes by updating the \"Last updated\" date of these Legal Terms, and you waive any right to receive specific notice of each such change. It is your responsibility to periodically review these Legal Terms to stay informed of updates. You will be subject to, and will be deemed to have been made aware of and to have accepted, the changes in any revised Legal Terms by your continued use of the Services after the date such revised Legal Terms are posted.
"},{"location":"terms/#1-our-services","title":"1. Our services","text":"The information provided when using the Services is not intended for distribution to or use by any person or entity in any jurisdiction or country where such distribution or use would be contrary to law or regulation or which would subject us to any registration requirement within such jurisdiction or country. Accordingly, those persons who choose to access the Services from other locations do so on their own initiative and are solely responsible for compliance with local laws, if and to the extent local laws are applicable.
The Services are not tailored to comply with industry-specific regulations (Health Insurance Portability and Accountability Act (HIPAA), Federal Information Security Management Act (FISMA), etc.), so if your interactions would be subjected to such laws, you may not use the Services. You may not use the Services in a way that would violate the Gramm-Leach-Bliley Act (GLBA).
"},{"location":"terms/#2-intelliectual-property-rights","title":"2. Intelliectual property rights","text":"Our intellectual property
We are the owner or the licensee of all intellectual property rights in our Services, including all source code, databases, functionality, software, website designs, audio, video, text, photographs, and graphics in the Services ( collectively, the \"Content\"), as well as the trademarks, service marks, and logos contained therein (the \"Marks\").
Our Content and Marks are protected by copyright and trademark laws (and various other intellectual property rights and unfair competition laws) and treaties in the United States and around the world.
The Content and Marks are provided in or through the Services \"AS IS\" for your personal, non-commercial use or internal business purpose only.
Your use of our Services
Subject to your compliance with these Legal Terms, including the \"Prohibited activities\" section below, we grant you a non-exclusive, non-transferable, revocable license to:
Except as set out in this section or elsewhere in our Legal Terms, no part of the Services and no Content or Marks may be copied, reproduced, aggregated, republished, uploaded, posted, publicly displayed, encoded, translated, transmitted, distributed, sold, licensed, or otherwise exploited for any commercial purpose whatsoever, without our express prior written permission.
If you wish to make any use of the Services, Content, or Marks other than as set out in this section or elsewhere in our Legal Terms, please address your request to: hello@dstack.ai. If we ever grant you the permission to post, reproduce, or publicly display any part of our Services or Content, you must identify us as the owners or licensors of the Services, Content, or Marks and ensure that any copyright or proprietary notice appears or is visible on posting, reproducing, or displaying our Content.
We reserve all rights not expressly granted to you in and to the Services, Content, and Marks.
Any breach of these Intellectual Property Rights will constitute a material breach of our Legal Terms and your right to use our Services will terminate immediately.
Your submissions
Please review this section and the \"Prohibited activities\" section carefully prior to using our Services to understand the (a) rights you give us and (b) obligations you have when you post or upload any content through the Services.
Submissions: By directly sending us any question, comment, suggestion, idea, feedback, or other information about the Services (\"Submissions\"), you agree to assign to us all intellectual property rights in such Submission. You agree that we shall own this Submission and be entitled to its unrestricted use and dissemination for any lawful purpose, commercial or otherwise, without acknowledgment or compensation to you.
You are responsible for what you post or upload: By sending us Submissions through any part of the Services you: * confirm that you have read and agree with our \"Prohibited activities\" and will not post, send, publish, upload, or * transmit through the Services any Submission that is illegal, harassing, hateful, harmful, defamatory, obscene, * bullying, abusive, discriminatory, threatening to any person or group, sexually explicit, false, inaccurate, deceitful, or misleading; * to the extent permissible by applicable law, waive any and all moral rights to any such Submission; * warrant that any such Submission are original to you or that you have the necessary rights and licenses to submit such Submissions and that you have full authority to grant us the above-mentioned rights in relation to your Submissions; and * warrant and represent that your Submissions do not constitute confidential information.
You are solely responsible for your Submissions and you expressly agree to reimburse us for any and all losses that we may suffer because of your breach of (a) this section, (b) any third party\u2019s intellectual property rights, or (c) applicable law.
"},{"location":"terms/#3-user-representations","title":"3. User representations","text":"By using the Services, you represent and warrant that: (1) all registration information you submit will be true, accurate, current, and complete; (2) you will maintain the accuracy of such information and promptly update such registration information as necessary; (3) you have the legal capacity and you agree to comply with these Legal Terms; ( 4) you are not a minor in the jurisdiction in which you reside; (5) you will not access the Services through automated or non-human means, whether through a bot, script or otherwise; (6) you will not use the Services for any illegal or unauthorized purpose; and (7) your use of the Services will not violate any applicable law or regulation.
If you provide any information that is untrue, inaccurate, not current, or incomplete, we have the right to suspend or terminate your account and refuse any and all current or future use of the Services (or any portion thereof).
"},{"location":"terms/#4-user-registration","title":"4. User registration","text":"You may be required to register to use the Services. You agree to keep your password confidential and will be responsible for all use of your account and password. We reserve the right to remove, reclaim, or change a username you select if we determine, in our sole discretion, that such username is inappropriate, obscene, or otherwise objectionable.
"},{"location":"terms/#5-purchases-and-payment","title":"5. Purchases and payment","text":"We accept the following forms of payment:
You agree to provide current, complete, and accurate purchase and account information for all purchases made via the Services. You further agree to promptly update account and payment information, including email address, payment method, and payment card expiration date, so that we can complete your transactions and contact you as needed. Sales tax will be added to the price of purchases as deemed required by us. We may change prices at any time. All payments shall be in US dollars.
You agree to pay all charges at the prices then in effect for your purchases and any applicable shipping fees, and you authorize us to charge your chosen payment provider for any such amounts upon placing your order. We reserve the right to correct any errors or mistakes in pricing, even if we have already requested or received payment.
We reserve the right to refuse any order placed through the Services. We may, in our sole discretion, limit or cancel quantities purchased per person, per household, or per order. These restrictions may include orders placed by or under the same customer account, the same payment method, and/or orders that use the same billing or shipping address. We reserve the right to limit or prohibit orders that, in our sole judgment, appear to be placed by dealers, resellers, or distributors.
"},{"location":"terms/#6-subscriptions","title":"6. Subscriptions","text":"Billing and Renewal
e.g. by topping up their balance manually using their credit card.
Cancellation
You can cancel your subscription at any time by contacting us using the contact information provided below. Your cancellation will take effect at the end of the current paid term. If you have any questions or are unsatisfied with our Services, please email us at hello@dstack.ai .
Fee Changes
We may, from time to time, make changes to the subscription fee and will communicate any price changes to you in accordance with applicable law.
"},{"location":"terms/#7-software","title":"7. Software","text":"We may include software for use in connection with our Services. If such software is accompanied by an end user license agreement (\"EULA\"), the terms of the EULA will govern your use of the software. If such software is not accompanied by a EULA, then we grant to you a non-exclusive, revocable, personal, and non-transferable license to use such software solely in connection with our services and in accordance with these Legal Terms. Any software and any related documentation is provided \"AS IS\" without warranty of any kind, either express or implied, including, without limitation, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. You accept any and all risk arising out of use or performance of any software. You may not reproduce or redistribute any software except in accordance with the EULA or these Legal Terms.
"},{"location":"terms/#8-prohibited-activities","title":"8. Prohibited activities","text":"You may not access or use the Services for any purpose other than that for which we make the Services available. The Services may not be used in connection with any commercial endeavors except those that are specifically endorsed or approved by us.
As a user of the Services, you agree not to:
The Services does not offer users to submit or post content.
"},{"location":"terms/#10-contribution-license","title":"10. Contribution license","text":"You and Services agree that we may access, store, process, and use any information and personal data that you provide following the terms of the Privacy Policy and your choices (including settings).
By submitting suggestions or other feedback regarding the Services, you agree that we can use and share such feedback for any purpose without compensation to you.
"},{"location":"terms/#11-social-media","title":"11. Social media","text":"As part of the functionality of the Services, you may link your account with online accounts you have with third-party service providers (each such account, a \"Third-Party Account\") by either: (1) providing your Third-Party Account login information through the Services; or (2) allowing us to access your Third-Party Account, as is permitted under the applicable terms and conditions that govern your use of each Third-Party Account. You represent and warrant that you are entitled to disclose your Third-Party Account login information to us and/or grant us access to your Third-Party Account, without breach by you of any of the terms and conditions that govern your use of the applicable Third-Party Account, and without obligating us to pay any fees or making us subject to any usage limitations imposed by the third-party service provider of the Third-Party Account. By granting us access to any Third-Party Accounts, you understand that (1) we may access, make available, and store (if applicable) any content that you have provided to and stored in your Third-Party Account (the \"Social Network Content\") so that it is available on and through the Services via your account, including without limitation any friend lists and (2) we may submit to and receive from your Third-Party Account additional information to the extent you are notified when you link your account with the Third-Party Account. Depending on the Third-Party Accounts you choose and subject to the privacy settings that you have set in such Third-Party Accounts, personally identifiable information that you post to your Third-Party Accounts may be available on and through your account on the Services. Please note that if a Third-Party Account or associated service becomes unavailable or our access to such Third-Party Account is terminated by the third-party service provider, then Social Network Content may no longer be available on and through the Services. You will have the ability to disable the connection between your account on the Services and your Third-Party Accounts at any time. PLEASE NOTE THAT YOUR RELATIONSHIP WITH THE THIRD-PARTY SERVICE PROVIDERS ASSOCIATED WITH YOUR THIRD-PARTY ACCOUNTS IS GOVERNED SOLELY BY YOUR AGREEMENT(S) WITH SUCH THIRD-PARTY SERVICE PROVIDERS. We make no effort to review any Social Network Content for any purpose, including but not limited to, for accuracy, legality, or non-infringement, and we are not responsible for any Social Network Content. You acknowledge and agree that we may access your email address book associated with a Third-Party Account and your contacts list stored on your mobile device or tablet computer solely for purposes of identifying and informing you of those contacts who have also registered to use the Services. You can deactivate the connection between the Services and your Third-Party Account by contacting us using the contact information below or through your account settings (if applicable). We will attempt to delete any information stored on our servers that was obtained through such Third-Party Account, except the username and profile picture that become associated with your account.
"},{"location":"terms/#12-third-party-websites-and-content","title":"12. Third-party websites and content","text":"The Services may contain (or you may be sent via the Site) links to other websites (\"Third-Party Websites\") as well as articles, photographs, text, graphics, pictures, designs, music, sound, video, information, applications, software, and other content or items belonging to or originating from third parties (\"Third-Party Content\"). Such Third-Party Websites and Third-Party Content are not investigated, monitored, or checked for accuracy, appropriateness, or completeness by us, and we are not responsible for any Third-Party Websites accessed through the Services or any Third-Party Content posted on, available through, or installed from the Services, including the content, accuracy, offensiveness, opinions, reliability, privacy practices, or other policies of or contained in the Third-Party Websites or the Third-Party Content. Inclusion of, linking to, or permitting the use or installation of any Third-Party Websites or any Third-Party Content does not imply approval or endorsement thereof by us. If you decide to leave the Services and access the Third-Party Websites or to use or install any Third-Party Content, you do so at your own risk, and you should be aware these Legal Terms no longer govern. You should review the applicable terms and policies, including privacy and data gathering practices, of any website to which you navigate from the Services or relating to any applications you use or install from the Services. Any purchases you make through Third-Party Websites will be through other websites and from other companies, and we take no responsibility whatsoever in relation to such purchases which are exclusively between you and the applicable third party. You agree and acknowledge that we do not endorse the products or services offered on Third-Party Websites and you shall hold us blameless from any harm caused by your purchase of such products or services. Additionally, you shall hold us blameless from any losses sustained by you or harm caused to you relating to or resulting in any way from any Third-Party Content or any contact with Third-Party Websites.
"},{"location":"terms/#13-services-management","title":"13. Services management","text":"We reserve the right, but not the obligation, to: (1) monitor the Services for violations of these Legal Terms; (2) take appropriate legal action against anyone who, in our sole discretion, violates the law or these Legal Terms, including without limitation, reporting such user to law enforcement authorities; (3) in our sole discretion and without limitation, refuse, restrict access to, limit the availability of, or disable (to the extent technologically feasible) any of your Contributions or any portion thereof; (4) in our sole discretion and without limitation, notice, or liability, to remove from the Services or otherwise disable all files and content that are excessive in size or are in any way burdensome to our systems; and (5) otherwise manage the Services in a manner designed to protect our rights and property and to facilitate the proper functioning of the Services.
"},{"location":"terms/#14-privacy-policy","title":"14. Privacy policy","text":"We care about data privacy and security. Please review our Privacy Policy. By using the Services, you agree to be bound by our Privacy Policy, which is incorporated into these Legal Terms. Please be advised the Services are hosted in Germany and United States. If you access the Services from any other region of the world with laws or other requirements governing personal data collection, use, or disclosure that differ from applicable laws in Germany and United States, then through your continued use of the Services, you are transferring your data to Germany and United States, and you expressly consent to have your data transferred to and processed in Germany and United States.
"},{"location":"terms/#15-term-and-termination","title":"15. Term and termination","text":"These Legal Terms shall remain in full force and effect while you use the Services. WITHOUT LIMITING ANY OTHER PROVISION OF THESE LEGAL TERMS, WE RESERVE THE RIGHT TO, IN OUR SOLE DISCRETION AND WITHOUT NOTICE OR LIABILITY, DENY ACCESS TO AND USE OF THE SERVICES (INCLUDING BLOCKING CERTAIN IP ADDRESSES), TO ANY PERSON FOR ANY REASON OR FOR NO REASON, INCLUDING WITHOUT LIMITATION FOR BREACH OF ANY REPRESENTATION, WARRANTY, OR COVENANT CONTAINED IN THESE LEGAL TERMS OR OF ANY APPLICABLE LAW OR REGULATION. WE MAY TERMINATE YOUR USE OR PARTICIPATION IN THE SERVICES OR DELETE YOUR ACCOUNT AND ANY CONTENT OR INFORMATION THAT YOU POSTED AT ANY TIME, WITHOUT WARNING, IN OUR SOLE DISCRETION.
If we terminate or suspend your account for any reason, you are prohibited from registering and creating a new account under your name, a fake or borrowed name, or the name of any third party, even if you may be acting on behalf of the third party. In addition to terminating or suspending your account, we reserve the right to take appropriate legal action, including without limitation pursuing civil, criminal, and injunctive redress.
"},{"location":"terms/#16-modifications-and-interruptions","title":"16. Modifications and interruptions","text":"We reserve the right to change, modify, or remove the contents of the Services at any time or for any reason at our sole discretion without notice. However, we have no obligation to update any information on our Services. We will not be liable to you or any third party for any modification, price change, suspension, or discontinuance of the Services.
We cannot guarantee the Services will be available at all times. We may experience hardware, software, or other problems or need to perform maintenance related to the Services, resulting in interruptions, delays, or errors. We reserve the right to change, revise, update, suspend, discontinue, or otherwise modify the Services at any time or for any reason without notice to you. You agree that we have no liability whatsoever for any loss, damage, or inconvenience caused by your inability to access or use the Services during any downtime or discontinuance of the Services. Nothing in these Legal Terms will be construed to obligate us to maintain and support the Services or to supply any corrections, updates, or releases in connection therewith.
"},{"location":"terms/#17-governing-law","title":"17. Governing law","text":"These Legal Terms are governed by and interpreted following the laws of Germany, and the use of the United Nations Convention of Contracts for the International Sales of Goods is expressly excluded. If your habitual residence is in the EU, and you are a consumer, you additionally possess the protection provided to you by obligatory provisions of the law in your country to residence. dstack GmbH and yourself both agree to submit to the non-exclusive jurisdiction of the courts of Bayern, which means that you may make a claim to defend your consumer protection rights in regards to these Legal Terms in Germany, or in the EU country in which you reside.
"},{"location":"terms/#18-dispute-resolution","title":"18. Dispute resolution","text":"Informal Negotiations
To expedite resolution and control the cost of any dispute, controversy, or claim related to these Legal Terms (each a \" Dispute\" and collectively, the \"Disputes\") brought by either you or us (individually, a \"Party\" and collectively, the \" Parties\"), the Parties agree to first attempt to negotiate any Dispute (except those Disputes expressly provided below) informally for at least thirty (30) days before initiating arbitration. Such informal negotiations commence upon written notice from one Party to the other Party.
Binding Arbitration
Any dispute arising from the relationships between the Parties to these Legal Terms shall be determined by one arbitrator who will be chosen in accordance with the Arbitration and Internal Rules of the European Court of Arbitration being part of the European Centre of Arbitration having its seat in Strasbourg, and which are in force at the time the application for arbitration is filed, and of which adoption of this clause constitutes acceptance. The seat of arbitration shall be Munich , Germany . The language of the proceedings shall be German . Applicable rules of substantive law shall be the law of Germany .
Restrictions
The Parties agree that any arbitration shall be limited to the Dispute between the Parties individually. To the full extent permitted by law, (a) no arbitration shall be joined with any other proceeding; (b) there is no right or authority for any Dispute to be arbitrated on a class-action basis or to utilize class action procedures; and (c) there is no right or authority for any Dispute to be brought in a purported representative capacity on behalf of the general public or any other persons.
Exceptions to Informal Negotiations and Arbitration
The Parties agree that the following Disputes are not subject to the above provisions concerning informal negotiations binding arbitration: (a) any Disputes seeking to enforce or protect, or concerning the validity of, any of the intellectual property rights of a Party; (b) any Dispute related to, or arising from, allegations of theft, piracy, invasion of privacy, or unauthorized use; and (c) any claim for injunctive relief. If this provision is found to be illegal or unenforceable, then neither Party will elect to arbitrate any Dispute falling within that portion of this provision found to be illegal or unenforceable and such Dispute shall be decided by a court of competent jurisdiction within the courts listed for jurisdiction above, and the Parties agree to submit to the personal jurisdiction of that court.
"},{"location":"terms/#19-corrections","title":"19. Corrections","text":"There may be information on the Services that contains typographical errors, inaccuracies, or omissions, including descriptions, pricing, availability, and various other information. We reserve the right to correct any errors, inaccuracies, or omissions and to change or update the information on the Services at any time, without prior notice.
"},{"location":"terms/#20-disclaimer","title":"20. Disclaimer","text":"THE SERVICES ARE PROVIDED ON AN AS-IS AND AS-AVAILABLE BASIS. YOU AGREE THAT YOUR USE OF THE SERVICES WILL BE AT YOUR SOLE RISK. TO THE FULLEST EXTENT PERMITTED BY LAW, WE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, IN CONNECTION WITH THE SERVICES AND YOUR USE THEREOF, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT. WE MAKE NO WARRANTIES OR REPRESENTATIONS ABOUT THE ACCURACY OR COMPLETENESS OF THE SERVICES' CONTENT OR THE CONTENT OF ANY WEBSITES OR MOBILE APPLICATIONS LINKED TO THE SERVICES AND WE WILL ASSUME NO LIABILITY OR RESPONSIBILITY FOR ANY (1) ERRORS, MISTAKES, OR INACCURACIES OF CONTENT AND MATERIALS, (2) PERSONAL INJURY OR PROPERTY DAMAGE, OF ANY NATURE WHATSOEVER, RESULTING FROM YOUR ACCESS TO AND USE OF THE SERVICES, (3) ANY UNAUTHORIZED ACCESS TO OR USE OF OUR SECURE SERVERS AND/OR ANY AND ALL PERSONAL INFORMATION AND/OR FINANCIAL INFORMATION STORED THEREIN, (4) ANY INTERRUPTION OR CESSATION OF TRANSMISSION TO OR FROM THE SERVICES, (5) ANY BUGS, VIRUSES, TROJAN HORSES, OR THE LIKE WHICH MAY BE TRANSMITTED TO OR THROUGH THE SERVICES BY ANY THIRD PARTY, AND/OR (6) ANY ERRORS OR OMISSIONS IN ANY CONTENT AND MATERIALS OR FOR ANY LOSS OR DAMAGE OF ANY KIND INCURRED AS A RESULT OF THE USE OF ANY CONTENT POSTED, TRANSMITTED, OR OTHERWISE MADE AVAILABLE VIA THE SERVICES. WE DO NOT WARRANT, ENDORSE, GUARANTEE, OR ASSUME RESPONSIBILITY FOR ANY PRODUCT OR SERVICE ADVERTISED OR OFFERED BY A THIRD PARTY THROUGH THE SERVICES, ANY HYPERLINKED WEBSITE, OR ANY WEBSITE OR MOBILE APPLICATION FEATURED IN ANY BANNER OR OTHER ADVERTISING, AND WE WILL NOT BE A PARTY TO OR IN ANY WAY BE RESPONSIBLE FOR MONITORING ANY TRANSACTION BETWEEN YOU AND ANY THIRD-PARTY PROVIDERS OF PRODUCTS OR SERVICES. AS WITH THE PURCHASE OF A PRODUCT OR SERVICE THROUGH ANY MEDIUM OR IN ANY ENVIRONMENT, YOU SHOULD USE YOUR BEST JUDGMENT AND EXERCISE CAUTION WHERE APPROPRIATE.
"},{"location":"terms/#21-limitations-of-liability","title":"21. Limitations of liability","text":"IN NO EVENT WILL WE OR OUR DIRECTORS, EMPLOYEES, OR AGENTS BE LIABLE TO YOU OR ANY THIRD PARTY FOR ANY DIRECT, INDIRECT, CONSEQUENTIAL, EXEMPLARY, INCIDENTAL, SPECIAL, OR PUNITIVE DAMAGES, INCLUDING LOST PROFIT, LOST REVENUE, LOSS OF DATA, OR OTHER DAMAGES ARISING FROM YOUR USE OF THE SERVICES, EVEN IF WE HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. NOTWITHSTANDING ANYTHING TO THE CONTRARY CONTAINED HEREIN, OUR LIABILITY TO YOU FOR ANY CAUSE WHATSOEVER AND REGARDLESS OF THE FORM OF THE ACTION, WILL AT ALL TIMES BE LIMITED TO THE LESSER OF THE AMOUNT PAID, IF ANY, BY YOU TO US DURING THE zero (0) MONTH PERIOD PRIOR TO ANY CAUSE OF ACTION ARISING OR $0.00 USD. CERTAIN US STATE LAWS AND INTERNATIONAL LAWS DO NOT ALLOW LIMITATIONS ON IMPLIED WARRANTIES OR THE EXCLUSION OR LIMITATION OF CERTAIN DAMAGES. IF THESE LAWS APPLY TO YOU, SOME OR ALL OF THE ABOVE DISCLAIMERS OR LIMITATIONS MAY NOT APPLY TO YOU, AND YOU MAY HAVE ADDITIONAL RIGHTS.
"},{"location":"terms/#22-indemnification","title":"22. Indemnification","text":"You agree to defend, indemnify, and hold us harmless, including our subsidiaries, affiliates, and all of our respective officers, agents, partners, and employees, from and against any loss, damage, liability, claim, or demand, including reasonable attorneys\u2019 fees and expenses, made by any third party due to or arising out of: (1) use of the Services; (2) breach of these Legal Terms; (3) any breach of your representations and warranties set forth in these Legal Terms; (4) your violation of the rights of a third party, including but not limited to intellectual property rights; or (5) any overt harmful act toward any other user of the Services with whom you connected via the Services. Notwithstanding the foregoing, we reserve the right, at your expense, to assume the exclusive defense and control of any matter for which you are required to indemnify us, and you agree to cooperate, at your expense, with our defense of such claims. We will use reasonable efforts to notify you of any such claim, action, or proceeding which is subject to this indemnification upon becoming aware of it.
"},{"location":"terms/#23-user-data","title":"23. User data","text":"We will maintain certain data that you transmit to the Services for the purpose of managing the performance of the Services, as well as data relating to your use of the Services. Although we perform regular routine backups of data, you are solely responsible for all data that you transmit or that relates to any activity you have undertaken using the Services. You agree that we shall have no liability to you for any loss or corruption of any such data, and you hereby waive any right of action against us arising from any such loss or corruption of such data.
"},{"location":"terms/#24-electronic-communications-transactions-and-signatures","title":"24. Electronic communications, transactions, and signatures","text":"Visiting the Services, sending us emails, and completing online forms constitute electronic communications. You consent to receive electronic communications, and you agree that all agreements, notices, disclosures, and other communications we provide to you electronically, via email and on the Services, satisfy any legal requirement that such communication be in writing. YOU HEREBY AGREE TO THE USE OF ELECTRONIC SIGNATURES, CONTRACTS, ORDERS, AND OTHER RECORDS, AND TO ELECTRONIC DELIVERY OF NOTICES, POLICIES, AND RECORDS OF TRANSACTIONS INITIATED OR COMPLETED BY US OR VIA THE SERVICES. You hereby waive any rights or requirements under any statutes, regulations, rules, ordinances, or other laws in any jurisdiction which require an original signature or delivery or retention of non-electronic records, or to payments or the granting of credits by any means other than electronic means.
"},{"location":"terms/#25-california-users-and-residents","title":"25. California users and residents","text":"If any complaint with us is not satisfactorily resolved, you can contact the Complaint Assistance Unit of the Division of Consumer Services of the California Department of Consumer Affairs in writing at 1625 North Market Blvd., Suite N 112, Sacramento, California 95834 or by telephone at (800) 952-5210 or (916) 445-1254.
"},{"location":"terms/#26-miscellaneous","title":"26. Miscellaneous","text":"These Legal Terms and any policies or operating rules posted by us on the Services or in respect to the Services constitute the entire agreement and understanding between you and us. Our failure to exercise or enforce any right or provision of these Legal Terms shall not operate as a waiver of such right or provision. These Legal Terms operate to the fullest extent permissible by law. We may assign any or all of our rights and obligations to others at any time. We shall not be responsible or liable for any loss, damage, delay, or failure to act caused by any cause beyond our reasonable control. If any provision or part of a provision of these Legal Terms is determined to be unlawful, void, or unenforceable, that provision or part of the provision is deemed severable from these Legal Terms and does not affect the validity and enforceability of any remaining provisions. There is no joint venture, partnership, employment or agency relationship created between you and us as a result of these Legal Terms or use of the Services. You agree that these Legal Terms will not be construed against us by virtue of having drafted them. You hereby waive any and all defenses you may have based on the electronic form of these Legal Terms and the lack of signing by the parties hereto to execute these Legal Terms.
"},{"location":"terms/#27-contact-us","title":"27. Contact us","text":"In order to resolve a complaint regarding the Services or to receive further information regarding use of the Services, please contact us at hello@dstack.ai.
"},{"location":"blog/","title":"Blog","text":""},{"location":"blog/archive/say-goodbye-to-managed-notebooks/","title":"Say goodbye to managed notebooks","text":"Data science and ML tools have made significant advancements in recent years. This blog post aims to examine the advantages of cloud dev environments (CDE) for ML engineers and compare them with web-based managed notebooks.
"},{"location":"blog/archive/say-goodbye-to-managed-notebooks/#notebooks-are-here-to-stay","title":"Notebooks are here to stay","text":"Jupyter notebooks are instrumental for interactive work with data. They provide numerous advantages such as high interactivity, visualization support, remote accessibility, and effortless sharing.
Managed notebook platforms, like Google Colab and AWS SageMaker have become popular thanks to their easy integration with clouds. With pre-configured environments, managed notebooks remove the need to worry about infrastructure.
"},{"location":"blog/archive/say-goodbye-to-managed-notebooks/#reproducibility-challenge","title":"Reproducibility challenge","text":"As the code evolves, it needs to be converted into Python scripts and stored in Git for improved organization and version control. Notebooks alone cannot handle this task, which is why they must be a part of a developer environment that also supports Python scripts and Git.
The JupyterLab project attempts to address this by turning notebooks into an IDE by adding a file browser, terminal, and Git support.
"},{"location":"blog/archive/say-goodbye-to-managed-notebooks/#ides-get-equipped-for-ml","title":"IDEs get equipped for ML","text":"Recently, IDEs have improved in their ability to support machine learning. They have started to combine the benefits of traditional IDEs and managed notebooks.
IDEs have upgraded their remote capabilities, with better SSH support. Additionally, they now offer built-in support for editing notebooks.
Two popular IDEs, VS Code and PyCharm, have both integrated remote capabilities and seamless notebook editing features.
"},{"location":"blog/archive/say-goodbye-to-managed-notebooks/#the-rise-of-app-ecosystem","title":"The rise of app ecosystem","text":"Notebooks have been beneficial for their interactivity and sharing features. However, there are new alternatives like Streamlit and Gradio that allow developers to build data apps using Python code. These frameworks not only simplify app-building but also enhance reproducibility by integrating with Git.
Hugging Face Spaces, for example, is a popular tool today for sharing Streamlit and Gradio apps with others.
"},{"location":"blog/archive/say-goodbye-to-managed-notebooks/#say-hello-to-cloud-dev-environments","title":"Say hello to cloud dev environments!","text":"Remote development within IDEs is becoming increasingly popular, and as a result, cloud dev environments have emerged as a new concept. Various managed services, such as Codespaces and GitPod, offer scalable infrastructure while maintaining the familiar IDE experience.
One such open-source tool is dstack, which enables you to define your dev environment declaratively as code and run it on any cloud.
type: dev-environment\nbuild:\n - apt-get update\n - apt-get install -y ffmpeg\n - pip install -r requirements.txt\nide: vscode\nWith this tool, provisioning the required hardware, setting up the pre-built environment (no Docker is needed), and fetching your local code is automated.
$ dstack run .\n\n RUN CONFIGURATION USER PROJECT INSTANCE SPOT POLICY\n honest-jellyfish-1 .dstack.yml peter gcp a2-highgpu-1g on-demand\n\nStarting SSH tunnel...\n\nTo open in VS Code Desktop, use one of these link:\n vscode://vscode-remote/ssh-remote+honest-jellyfish-1/workflow\n\nTo exit, press Ctrl+C.\nYou can securely access the cloud development environment with the desktop IDE of your choice.
Learn more
Check out our guide for running dev environments in your cloud.
"},{"location":"blog/dstack-sky-own-cloud-accounts/","title":"dstack Sky now allows using your own cloud accounts","text":"dstack Sky enables you to access GPUs from the global marketplace at the most competitive rates. However, sometimes you may want to use your own cloud accounts. With today's release, both options are now supported.
"},{"location":"blog/dstack-sky-own-cloud-accounts/#configure-backends","title":"Configure backends","text":"To use your own cloud account, open the project settings and edit the corresponding backend.
You can configure your cloud accounts for any of the supported providers, including AWS, GCP, Azure, TensorDock, Lambda, CUDO, RunPod, and Vast.ai.
Additionally, you can disable certain backends if you do not plan to use them.
Typically, if you prefer using your own cloud accounts, it's recommended that you use the open-source version of dstack. However, if you prefer not to host it yourself, now you can use dstack Sky with your own cloud accounts as well.
Seeking the cheapest on-demand and spot cloud GPUs? dstack Sky has you covered!
Need help, have a question, or just want to stay updated?
Join Discord
"},{"location":"blog/dstack-sky/","title":"Introducing dstack Sky","text":"Today we're previewing dstack Sky, a service built on top of dstack that enables you to get GPUs at competitive rates from a wide pool of providers.
dstack's CLI and APIdstack is an open-source tool designed for managing AI infrastructure across various cloud platforms. It's lighter and more specifically geared towards AI tasks compared to Kubernetes.
Due to its support for multiple cloud providers, dstack is frequently used to access on-demand and spot GPUs across multiple clouds. From our users, we've learned that managing various cloud accounts, quotas, and billing can be cumbersome.
To streamline this process, we introduce dstack Sky, a managed service that enables users to access GPUs from multiple providers through dstack \u2013 without needing an account in each cloud provider.
Instead of running dstack server yourself, you point dstack config to a project set up with dstack Sky.
$ dstack config --url https://sky.dstack.ai \\\n --project my-awesome-project \\\n --token ca1ee60b-7b3f-8943-9a25-6974c50efa75\nNow, you can use dstack's CLI or API \u2013 just like you would with your own cloud accounts.
$ dstack run . -b tensordock -b vastai\n\n # BACKEND REGION RESOURCES SPOT PRICE \n 1 vastai canada 16xCPU/64GB/1xRTX4090/1TB no $0.35\n 2 vastai canada 16xCPU/64GB/1xRTX4090/400GB no $0.34\n 3 tensordock us 8xCPU/48GB/1xRTX4090/480GB no $0.74\n ...\n Shown 3 of 50 offers, $0.7424 max\n\nContinue? [y/n]:\nBackends
dstack Sky supports the same backends as the open-source version, except that you don't need to set them up. By default, it uses all supported backends.
You can use both on-demand and spot instances without needing to manage quotas, as they are automatically handled for you.
With dstack Sky you can use all of dstack's features, incl. dev environments, tasks, services, and fleets.
To use services, the open-source version requires setting up a gateway with your own domain. dstack Sky comes with a pre-configured gateway.
$ dstack gateway list\n BACKEND REGION NAME ADDRESS DOMAIN DEFAULT\n aws eu-west-1 dstack 3.252.79.143 my-awesome-project.sky.dstack.ai \u2713\nIf you run it with dstack Sky, the service's endpoint will be available at https://<run name>.<project name>.sky.dstack.ai.
Let's say we define a service:
type: service\n# Deploys Mixtral 8x7B with Ollama\n\n# Serve model using Ollama's Docker image\nimage: ollama/ollama\ncommands:\n - ollama serve &\n - sleep 3\n - ollama pull mixtral\n - fg\nport: 11434\n\n# Configure hardware requirements\nresources:\n gpu: 48GB..80GB\n\n# Enable OpenAI compatible endpoint\nmodel:\n type: chat\n name: mixtral\n format: openai\nIf it has a model mapping, the model will be accessible at https://gateway.<project name>.sky.dstack.ai via the OpenAI compatible interface.
from openai import OpenAI\n\n\nclient = OpenAI(\n base_url=\"https://gateway.<project name>.sky.dstack.ai\",\n api_key=\"<dstack token>\"\n)\n\ncompletion = client.chat.completions.create(\n model=\"mixtral\",\n messages=[\n {\"role\": \"user\", \"content\": \"Compose a poem that explains the concept of recursion in programming.\"}\n ]\n)\n\nprint(completion.choices[0].message)\nNow, you can choose \u2014 either use dstack via the open-source version or via dstack Sky, or even use them side by side.
Credits
Are you an active contributor to the AI community? Request free dstack Sky credits.
dstack Sky is live on Product Hunt. Support it by giving it your vote!
Join Discord
"},{"location":"changelog/","title":"Blog","text":""},{"location":"docs/","title":"What is dstack?","text":"dstack is an open-source container orchestration engine for AI. It accelerates the development, training, and deployment of AI models, and simplifies the management of clusters.
dstack is easy to use with any cloud or on-prem servers. Supported cloud providers include AWS, GCP, Azure, OCI, Lambda, TensorDock, Vast.ai, RunPod, and CUDO. For using dstack with on-prem servers, see fleets.
dstack supports NVIDIA GPU and Google Cloud TPU out of the box.
Before using dstack, install the server and configure backends for each cloud account (or Kubernetes cluster) that you intend to use.
dstack supports three types of run configurations:
Each type of run configuration allows you to specify commands for execution, required compute resources, retry policies, auto-scaling rules, authorization settings, and more.
Configuration can be defined as YAML files within your repo.
"},{"location":"docs/#2-run-configurations","title":"2. Run configurations","text":"Run any defined configuration either via dstack CLI or API.
dstack automatically handles provisioning, interruptions, port-forwarding, auto-scaling, network, volumes, run failures, out-of-capacity errors, and more.
Use fleets to provision and manage clusters and instances, both in the cloud and on-prem.
"},{"location":"docs/#where-do-i-start","title":"Where do I start?","text":"Before scheduling a task or deploying a model, you may want to run code interactively. Dev environments allow you to provision a remote machine set up with your code and favorite IDE with just one command.
"},{"location":"docs/dev-environments/#configuration","title":"Configuration","text":"First, create a YAML file in your project folder. Its name must end with .dstack.yml (e.g. .dstack.yml or dev.dstack.yml are both acceptable).
type: dev-environment\n\n# Specify the Python version, or your Docker image\npython: \"3.11\"\n\n# This pre-configures the IDE with required extensions\nide: vscode\n\n# Specify GPU, disk, and other resource requirements\nresources:\n gpu: 80GB\nIf you don't specify your Docker image, dstack uses the base image (pre-configured with Python, Conda, and essential CUDA drivers).
Reference
See the .dstack.yml reference for all supported configuration options and examples.
"},{"location":"docs/dev-environments/#running","title":"Running","text":"To run a configuration, use the dstack run command followed by the working directory path, configuration file path, and other options.
$ dstack run . -f .dstack.yml\n\n BACKEND REGION RESOURCES SPOT PRICE\n tensordock unitedkingdom 10xCPU, 80GB, 1xA100 (80GB) no $1.595\n azure westus3 24xCPU, 220GB, 1xA100 (80GB) no $3.673\n azure westus2 24xCPU, 220GB, 1xA100 (80GB) no $3.673\n\nContinue? [y/n]: y\n\nProvisioning `fast-moth-1`...\n---> 100%\n\nTo open in VS Code Desktop, use this link:\n vscode://vscode-remote/ssh-remote+fast-moth-1/workflow\nWhen dstack provisions the dev environment, it mounts the project folder contents.
If there are large files or folders you'd like to avoid uploading, you can list them in .gitignore.
By default, dstack run reuses idle instances from one of the existing fleets. If no idle instances meet the requirements, it creates a new fleet using one of the configured backends.
To have the fleet deleted after a certain idle time automatically, set termination_idle_time. By default, it's set to 5min.
Reference
See the CLI reference for more details on how dstack run works.
To open the dev environment in your desktop IDE, use the link from the output (such as vscode://vscode-remote/ssh-remote+fast-moth-1/workflow).
Alternatively, while the CLI is attached to the run, you can connect to the dev environment via SSH:
$ ssh fast-moth-1\nThe dstack ps command lists all running runs and their status.
Once the run exceeds the max duration, or when you use dstack stop, the dev environment and its cloud resources are deleted.
.dstack.yml reference for more details and examplesFleets enable efficient provisioning and management of clusters and instances, both in the cloud and on-prem. Once a fleet is created, it can be reused by dev environments, tasks, and services.
Fleets is a new feature. To use it, ensure you've installed version 0.18.7 or higher.
To create a fleet, create a YAML file in your project folder. Its name must end with .dstack.yml (e.g. .dstack.yml or fleet.dstack.yml are both acceptable).
To provision a fleet in the cloud using the configured backends, specify the required resources, number of nodes, and other optional parameters.
type: fleet\nname: my-fleet\nplacement: cluster\nnodes: 2\nresources:\n gpu: 24GB\nSet placement to cluster if the nodes should be interconnected (e.g. if you'd like to use them for multi-node tasks). In that case, dstack will provision all nodes in the same backend and region.
Defining fleets with YAML isn't supported yet for the kubernetes, vastai, and runpod backends.
To create a fleet from on-prem servers, specify their hosts along with the user, port, and SSH key for connection via SSH.
type: fleet\nname: my-fleet\nplacement: cluster\nssh_config:\n user: ubuntu\n identity_file: ~/.ssh/id_rsa\n hosts:\n - 3.255.177.51\n - 3.255.177.52\nRequirements
The on-prem servers should be pre-installed with CUDA 12.1 and NVIDIA Docker. The user should have sudo access.
Set placement to cluster if the nodes are interconnected (e.g. if you'd like to use them for multi-node tasks). In that case, by default, dstack will automatically detect the private network. You can specify the network parameter manually.
Reference
See the .dstack.yml reference for all supported configuration options and examples.
"},{"location":"docs/fleets/#creating-and-updating-fleets","title":"Creating and updating fleets","text":"To create or update the fleet, simply call the dstack apply command:
$ dstack apply -f my-gcp-fleet.dstack.yml\nFleet my-fleet does not exist yet. Create the fleet? [y/n]: y\n FLEET INSTANCE BACKEND RESOURCES PRICE STATUS CREATED \n my-fleet 0 pending now \n 1 pending now \nOnce the status of instances change to idle, they can be used by dstack run.
By default, dstack run tries to reuse idle instances from existing fleets. If no idle instances meet the requirements, dstack run creates a new fleet automatically. To avoid creating new fleet, specify pass --reuse to dstack run.
If you want a fleet to be automatically deleted after a certain idle time, you can set the you can set the termination_idle_time property.
The dstack fleet command lists fleet instances and theri status:
$ dstack fleet\n FLEET INSTANCE BACKEND GPU PRICE STATUS CREATED \n my-fleet 0 gcp (europe-west-1) L4:24GB (spot) $0.1624 idle 3 mins ago \n 1 gcp (europe-west-1) L4:24GB (spot) $0.1624 idle 3 mins ago \nWhen a fleet isn't used by run, you can delete it via dstack delete:
$ dstack delete -f cluster.dstack.yaml\nDelete the fleet my-gcp-fleet? [y/n]: y\nFleet my-gcp-fleet deleted\nYou can pass either the path to the configuration file or the fleet name directly.
To terminate and delete specific instances from a fleet, pass -i INSTANCE_NUM.
Before using dstack, install the server and configure backends.
To use dstack's CLI in a folder, first run dstack init within that folder.
$ mkdir quickstart && cd quickstart\n$ dstack init\nYour folder can be a regular local folder or a Git repo.
"},{"location":"docs/quickstart/#define-a-configuration","title":"Define a configuration","text":"Define what you want to run as a YAML file. The filename must end with .dstack.yml (e.g., .dstack.yml or train.dstack.yml are both acceptable).
Dev environments allow you to quickly provision a machine with a pre-configured environment, resources, IDE, code, etc.
type: dev-environment\n\n# Use either `python` or `image` to configure environment\npython: \"3.11\"\n# image: ghcr.io/huggingface/text-generation-inference:latest\n\nide: vscode\n\n# (Optional) Configure `gpu`, `memory`, `disk`, etc\nresources:\n gpu: 24GB\nTasks make it very easy to run any scripts, be it for training, data processing, or web apps. They allow you to pre-configure the environment, resources, code, etc.
type: task\n\npython: \"3.11\"\nenv:\n - HF_HUB_ENABLE_HF_TRANSFER=1\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py\n\n# (Optional) Configure `gpu`, `memory`, `disk`, etc\nresources:\n gpu: 24GB\nEnsure requirements.txt and train.py are in your folder. You can take them from examples.
Services make it easy to deploy models and apps cost-effectively as public endpoints, allowing you to use any frameworks.
type: service\n\nimage: ghcr.io/huggingface/text-generation-inference:latest\nenv:\n - HUGGING_FACE_HUB_TOKEN # required to run gated models\n - MODEL_ID=mistralai/Mistral-7B-Instruct-v0.1\ncommands:\n - text-generation-launcher --port 8000 --trust-remote-code\nport: 8000\n\n# (Optional) Configure `gpu`, `memory`, `disk`, etc\nresources:\n gpu: 24GB\nRun a configuration using the dstack run command, followed by the working directory path (e.g., .), and the path to the configuration file.
$ dstack run . -f train.dstack.yml\n\n BACKEND REGION RESOURCES SPOT PRICE\n tensordock unitedkingdom 10xCPU, 80GB, 1xA100 (80GB) no $1.595\n azure westus3 24xCPU, 220GB, 1xA100 (80GB) no $3.673\n azure westus2 24xCPU, 220GB, 1xA100 (80GB) no $3.673\n\nContinue? [y/n]: y\n\nProvisioning...\n---> 100%\n\nEpoch 0: 100% 1719/1719 [00:18<00:00, 92.32it/s, loss=0.0981, acc=0.969]\nEpoch 1: 100% 1719/1719 [00:18<00:00, 92.32it/s, loss=0.0981, acc=0.969]\nEpoch 2: 100% 1719/1719 [00:18<00:00, 92.32it/s, loss=0.0981, acc=0.969]\nThe dstack run command automatically uploads your code, including any local uncommitted changes.
Fleets
By default, dstack run reuses idle instances from one of the existing fleets. If no idle instances meet the requirements, it creates a new fleet using one of the configured backends.
Services make it easy to deploy models and web applications as public, secure, and scalable endpoints. They are provisioned behind a gateway that automatically provides an HTTPS domain, handles authentication, distributes load, and performs auto-scaling.
GatewaysIf you're using the open-source server, you must set up a gateway before you can run a service.
If you're using dstack Sky , the gateway is already set up for you.
"},{"location":"docs/services/#configuration","title":"Configuration","text":"First, create a YAML file in your project folder. Its name must end with .dstack.yml (e.g. .dstack.yml or serve.dstack.yml are both acceptable).
type: service\n\npython: \"3.11\"\nenv:\n - MODEL=NousResearch/Llama-2-7b-chat-hf\ncommands:\n - pip install vllm\n - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000\nport: 8000\n\nresources:\n gpu: 80GB\n\n# (Optional) Enable the OpenAI-compatible endpoint\nmodel:\n format: openai\n type: chat\n name: NousResearch/Llama-2-7b-chat-hf\nIf you don't specify your Docker image, dstack uses the base image (pre-configured with Python, Conda, and essential CUDA drivers).
Auto-scaling
By default, the service is deployed to a single instance. However, you can specify the number of replicas and scaling policy. In this case, dstack auto-scales it based on the load.
Reference
See the .dstack.yml reference for all supported configuration options and examples.
"},{"location":"docs/services/#running","title":"Running","text":"To run a configuration, use the dstack run command followed by the working directory path, configuration file path, and any other options.
$ dstack run . -f serve.dstack.yml\n\n BACKEND REGION RESOURCES SPOT PRICE\n tensordock unitedkingdom 10xCPU, 80GB, 1xA100 (80GB) no $1.595\n azure westus3 24xCPU, 220GB, 1xA100 (80GB) no $3.673\n azure westus2 24xCPU, 220GB, 1xA100 (80GB) no $3.673\n\nContinue? [y/n]: y\n\nProvisioning...\n---> 100%\n\nService is published at https://yellow-cat-1.example.com\nWhen deploying the service, dstack run mounts the current folder's contents.
If there are large files or folders you'd like to avoid uploading, you can list them in .gitignore.
By default, dstack run reuses idle instances from one of the existing fleets. If no idle instances meet the requirements, it creates a new fleet using one of the configured backends.
To have the fleet deleted after a certain idle time automatically, set termination_idle_time. By default, it's set to 5min.
Reference
See the CLI reference for more details on how dstack run works.
One the service is up, its endpoint is accessible at https://<run name>.<gateway domain>.
By default, the service endpoint requires the Authorization header with Bearer <dstack token>.
$ curl https://yellow-cat-1.example.com/v1/chat/completions \\\n -H 'Content-Type: application/json' \\\n -H 'Authorization: Bearer <dstack token>' \\\n -d '{\n \"model\": \"NousResearch/Llama-2-7b-chat-hf\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Compose a poem that explains the concept of recursion in programming.\"\n }\n ]\n }'\nAuthorization can be disabled by setting auth to false in the service configuration file.
In case the service has the model mapping configured, you will also be able to access the model at https://gateway.<gateway domain> via the OpenAI-compatible interface.
The dstack ps command lists all running runs and their status.
When you use dstack stop, the service and its cloud resources are deleted.
.dstack.yml reference for more details and examplesTasks allow for convenient scheduling of various batch jobs, such as training, fine-tuning, or data processing. They can also be used to run web applications when features offered by services are not needed, such as for debugging.
You can run tasks on a single machine or on a cluster of nodes.
"},{"location":"docs/tasks/#configuration","title":"Configuration","text":"First, create a YAML file in your project folder. Its name must end with .dstack.yml (e.g. .dstack.yml or train.dstack.yml are both acceptable).
type: task\n\npython: \"3.11\"\nenv:\n - HF_HUB_ENABLE_HF_TRANSFER=1\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - tensorboard --logdir results/runs &\n - python fine-tuning/qlora/train.py\nports:\n - 6000\n\n# (Optional) Configure `gpu`, `memory`, `disk`, etc\nresources:\n gpu: 80GB\nIf you don't specify your Docker image, dstack uses the base image (pre-configured with Python, Conda, and essential CUDA drivers).
Distributed tasks
By default, tasks run on a single instance. However, you can specify the number of nodes. In this case, dstack provisions a cluster of instances.
Reference
See the .dstack.yml reference for all supported configuration options and examples.
"},{"location":"docs/tasks/#running","title":"Running","text":"To run a configuration, use the dstack run command followed by the working directory path, configuration file path, and other options.
$ dstack run . -f train.dstack.yml\n\n BACKEND REGION RESOURCES SPOT PRICE\n tensordock unitedkingdom 10xCPU, 80GB, 1xA100 (80GB) no $1.595\n azure westus3 24xCPU, 220GB, 1xA100 (80GB) no $3.673\n azure westus2 24xCPU, 220GB, 1xA100 (80GB) no $3.673\n\nContinue? [y/n]: y\n\nProvisioning...\n---> 100%\n\nTensorBoard 2.13.0 at http://localhost:6006/ (Press CTRL+C to quit)\n\nEpoch 0: 100% 1719/1719 [00:18<00:00, 92.32it/s, loss=0.0981, acc=0.969]\nEpoch 1: 100% 1719/1719 [00:18<00:00, 92.32it/s, loss=0.0981, acc=0.969]\nEpoch 2: 100% 1719/1719 [00:18<00:00, 92.32it/s, loss=0.0981, acc=0.969]\nIf the task specifies ports, dstack run automatically forwards them to your local machine for convenient and secure access.
When running the task, dstack run mounts the current folder's contents.
If there are large files or folders you'd like to avoid uploading, you can list them in .gitignore.
By default, dstack run reuses idle instances from one of the existing fleets. If no idle instances meet the requirements, it creates a new fleet using one of the configured backends.
To have the fleet deleted after a certain idle time automatically, set termination_idle_time. By default, it's set to 5min.
Reference
See the CLI reference for more details on how dstack run works.
The dstack ps command lists all running runs and their status.
Once you use dstack stop (or when the run exceeds the max_duration), the instances return to the fleet.
.dstack.yml reference for more details and examplesGateways handle the ingress traffic of running services. They provide services with HTTPS domains, handle authentication, distribute load, and perform auto-scaling. In order to run a service, you need to have at least one gateway set up.
If you're using dstack Sky , the gateway is already set up for you.
"},{"location":"docs/concepts/gateways/#configuration","title":"Configuration","text":"First, create a YAML file in your project folder. Its name must end with .dstack.yml (e.g. .dstack.yml or gateway.dstack.yml are both acceptable).
type: gateway\nname: example-gateway\n\nbackend: aws\nregion: eu-west-1\ndomain: example.com\nA domain name is required to create a gateway.
Reference
See the .dstack.yml reference for all supported configuration options and examples.
"},{"location":"docs/concepts/gateways/#creating-and-updating-gateways","title":"Creating and updating gateways","text":"To create or update the gateway, simply call the dstack apply command:
$ dstack apply . -f examples/deployment/gateway.dstack.yml\n\nThe example-gateway doesn't exist. Create it? [y/n]: y\n\n BACKEND REGION NAME HOSTNAME DOMAIN DEFAULT STATUS\n aws eu-west-1 example-gateway example.com \u2713 submitted\nOnce the gateway is assigned a hostname, go to your domain's DNS settings and add an A DNS record for *.<gateway domain> (e.g., *.example.com) pointing to the gateway's hostname.
This will allow you to access runs and models using this domain.
"},{"location":"docs/concepts/gateways/#managing-gateways","title":"Managing gateways","text":""},{"location":"docs/concepts/gateways/#listing-gateways","title":"Listing gateways","text":"The dstack gateway list command lists existing gateways and their status.
To delete a gateway, pass gateway configuration to dstack delete:
$ dstack delete . -f examples/deployment/gateway.dstack.yml\n.dstack.yml reference for more details and examplesPools enable the efficient reuse of cloud instances and on-premises servers across runs, simplifying their management.
"},{"location":"docs/concepts/pools/#adding-instances","title":"Adding instances","text":""},{"location":"docs/concepts/pools/#automatic-provisioning","title":"Automatic provisioning","text":"By default, when using the dstack run command, it tries to reuse an instance from a pool. If no idle instance meets the requirements, dstack automatically provisions a new cloud instance and adds it to the pool.
To avoid provisioning new cloud instances with dstack run, use --reuse. Your run will be assigned to an idle instance in the pool. If there are no available idle instances in the pool, the run will fail.
By default, dstack run sets the idle duration of a newly provisioned instance to 5m. This means that if the run is finished and the instance remains idle for longer than five minutes, it is automatically removed from the pool. To override the default idle duration, use --idle-duration DURATION with dstack run.
To manually provision a cloud instance and add it to a pool, use dstack pool add:
$ dstack pool add --gpu 80GB\n\n BACKEND REGION RESOURCES SPOT PRICE\n tensordock unitedkingdom 10xCPU, 80GB, 1xA100 (80GB) no $1.595\n azure westus3 24xCPU, 220GB, 1xA100 (80GB) no $3.673\n azure westus2 24xCPU, 220GB, 1xA100 (80GB) no $3.673\n\nContinue? [y/n]: y\nThe dstack pool add command allows specifying resource requirements, along with the spot policy, idle duration, max price, retry policy, and other policies.
The default idle duration if you're using dstack pool add is 72h. To override it, use the --idle-duration DURATION argument.
You can also specify the policies via .dstack/profiles.yml instead of passing them as arguments. For more details on policies and their defaults, refer to .dstack/profiles.yml.
The dstack pool add command is not supported for Kubernetes, VastAI, and RunPod backends yet.
Any on-prem server that can be accessed via SSH can be added to a pool and used to run workloads.
To add on-prem servers to the pool, use the dstack pool add-ssh command and pass the hostname of your server along with the SSH key.
$ dstack pool add-ssh -i ~/.ssh/id_rsa ubuntu@54.73.155.119\nThe command accepts the same arguments as the standard ssh command.
Requirements
The on-prem server should be pre-installed with CUDA 12.1 and NVIDIA Docker.
Once the instance is provisioned, you'll see it in the pool and will be able to run workloads on it.
"},{"location":"docs/concepts/pools/#clusters","title":"Clusters","text":"If you want on-prem instances to run multi-node tasks, ensure these on-prem servers share the same private network. Additionally, you need to pass the --network option to dstack pool add-ssh:
$ dstack pool add-ssh -i ~/.ssh/id_rsa ubuntu@54.73.155.119 \\\n --network 10.0.0.0/24\nThe --network argument accepts the IP address range (CIDR) of the private network of the instance.
Once you've added multiple instances with the same network value, you can use them as a cluster to run multi-node tasks.
"},{"location":"docs/concepts/pools/#removing-instances","title":"Removing instances","text":"If the instance remains idle for the configured idle duration, dstack removes it and deletes all cloud resources.
To remove an instance from the pool manually, use the dstack pool rm command.
$ dstack pool rm <instance name>\nThe dstack pool ps command lists active instances and their status (busy or idle).
Volumes allow you to persist data between runs. dstack simplifies managing volumes and lets you mount them to a specific directory when working with dev environments, tasks, and services.
Experimental
Volumes are currently experimental and only work with the aws and runpod backends. Support for other backends is coming soon.
First, create a YAML file in your project folder. Its name must end with .dstack.yml (e.g. .dstack.yml or vol.dstack.yml are both acceptable).
type: volume\nname: my-new-volume\nbackend: aws\nregion: eu-central-1\nsize: 100GB\nIf you use this configuration, dstack will create a new volume based on the specified options.
Registering existing volumes
If you prefer not to create a new volume but to reuse an existing one (e.g., created manually), you can specify its ID via volume_id. In this case, dstack will register the specified volume so that you can use it with dev environments, tasks, and services.
Reference
See the .dstack.yml reference for all supported configuration options and examples.
"},{"location":"docs/concepts/volumes/#creating-and-registering-volumes","title":"Creating and registering volumes","text":"To create or register the volume, simply call the dstack apply command:
$ dstack apply -f volume.dstack.yml\nVolume my-new-volume does not exist yet. Create the volume? [y/n]: y\n NAME BACKEND REGION STATUS CREATED \n my-new-volume aws eu-central-1 submitted now \nWhen creating the volume dstack automatically creates an ext4 file system on it.
Once created, the volume can be attached with dev environments, tasks, and services.
"},{"location":"docs/concepts/volumes/#attaching-volumes","title":"Attaching volumes","text":"Dev environments, tasks, and services let you attach any number of volumes. To attach a volume, simply specify its name using the volumes property and specify where to mount its contents:
type: dev-environment\nide: vscode\nvolumes:\n - name: my-new-volume\n path: /volume_data\nOnce you run this configuration, the contents of the volume will be attached to /volume_data inside the dev environment, and its contents will persist across runs.
Limitations
When you're running a dev environment, task, or service with dstack, it automatically mounts the project folder contents to /workflow (and sets that as the current working directory). Right now, dstack doesn't allow you to attach volumes to /workflow or any of its subdirectories.
The dstack volume list command lists created and registered volumes:
$ dstack volume list\nNAME BACKEND REGION STATUS CREATED\n my-new-volume aws eu-central-1 active 3 weeks ago\nWhen the volume isn't attached to any active dev environment, task, or service, you can delete it using dstack delete:
$ dstack delete -f vol.dstack.yaml\nIf the volume was created using dstack, it will be physically destroyed along with the data. If you've registered an existing volume, it will be de-registered with dstack but will keep the data.
Since volumes are backed up by cloud network disks, you can only use them within the same cloud. If you need to access data across different backends, you should either use object storage or replicate the data across multiple volumes.
Using volumes across regionsTypically, network volumes are associated with specific regions, so you can't use them in other regions. Often, volumes are also linked to availability zones, but some providers support volumes that can be used across different availability zones within the same region.
Attaching volumes to multiple runs and instancesYou can mount a volume in multiple runs. This feature is currently supported only by the runpod backend.
Below are tips and tricks to use dstack more efficiently.
Before running a task or service, it's recommended that you first start with a dev environment. Dev environments allow you to run commands interactively.
Once the commands work, go ahead and run them as a task or a service.
NotebooksVS Code
When you access a dev environment using your desktop VS Code, it allows you to work with Jupyter notebooks via its pre-configured and easy-to-use extension.
JupyterLab
If you prefer to use JupyterLab, you can run it as a task:
type: task\n\ncommands:\n - pip install jupyterlab\n - jupyter lab --allow-root\n\nports:\n - 8888\nTasks can be used not only for batch jobs but also for web applications.
type: task\n\npython: \"3.11\"\n\ncommands:\n - pip3 install streamlit\n - streamlit hello\n\nports: \n - 8501\nWhile you run a task, dstack forwards the remote ports to localhost.
$ dstack run . -f app.dstack.yml\n\n Welcome to Streamlit. Check out our demo in your browser.\n\n Local URL: http://localhost:8501\nThis allows you to access the remote 8501 port on localhost:8501 while the CLI is attached.
If you want to override the local port, use the --port option:
$ dstack run . -f app.dstack.yml --port 3000:8501\nThis will forward the remote 8501 port to localhost:3000.
Services require a gateway but they also provide additional features for production-grade service deployment not offered by tasks, such as HTTPS domains and auto-scaling. If you run a web app as a task and it works, go ahead and run it as a service.
"},{"location":"docs/guides/protips/#environment-variables","title":"Environment variables","text":"If a configuration requires an environment variable that you don't want to hardcode in the YAML, you can define it without assigning a value:
type: dev-environment\n\nenv:\n - HUGGING_FACE_HUB_TOKEN\n\npython: \"3.11\"\nide: vscode\nThen, you can pass the environment variable either via the shell:
HUGGING_FACE_HUB_TOKEN=... dstack run . -f .dstack.yml\nOr via the -e option of the dstack run command:
dstack run . -f .dstack.yml -e HUGGING_FACE_HUB_TOKEN=...\nA better way to configure environment variables not hardcoded in YAML is by specifying them in a .env file:
HUGGING_FACE_HUB_TOKEN=...\nIf you install direnv , it will automatically pass the environment variables from the .env file to the dstack run command.
Remember to add .env to .gitignore to avoid pushing it to the repo.
dstack has support for volumes to persist data across different runs and instance interruptions. Volumes are ideal for storing intermediate work and data that should be quickly accessible.
You can also load and save data using an object storage like S3 or HuggingFace Datasets. For models, it's best to use services like HuggingFace Hub. dstack has no explicit support for object storage. You can load and save data directly from your code.
By default, the dstack run command reuses an idle instance from the pool. If no instance matches the requirements, it creates a new one.
When the run finishes, the instance remains idle for the configured time (by default, 5m) before it gets destroyed.
You can change the default idle duration by using --idle-duration DURATION with dstack run, or set termination_idle_duration in the configuration or profile.
An idle instance can be destroyed at any time via dstack pool rm INSTANCE_NAME.
If you don't want to specify the same parameters for each configuration, you can define them once via profiles and reuse them across configurations.
This can be handy, for example, for configuring parameters such as max_duration, max_price, termination_idle_duration, regions, etc.
Set default to true in your profile, and it will be applied automatically to any run.
By default, dstack run runs in attached mode. This means it streams the logs as they come in and, in the case of a task, forwards its ports to localhost.
If you detach the CLI, you can re-attach it using dstack logs -a RUN_NAME.
To run in detached mode, use -d with dstack run.
dstack natively supports NVIDIA GPU, and Google Cloud TPU accelerator chips.
The gpu property withing resources (or the --gpu option with dstack run) allows specifying not only memory size but also GPU names, their memory, and quantity.
Examples:
1 (any GPU)A100 (A100)24GB.. (any GPU starting from 24GB)24GB..40GB:2 (two GPUs between 24GB and 40GB)A10G,A100 (either A10G or A100)A100:80GB (one A100 of 80GB)A100:2 (two A100)A100:40GB:2 (two A100 40GB)tpu-v2-8 (v2 with 8 TPU cores)Currently, you can't specify other than 8 TPU cores. This means only single host workloads are supported. Support for multiple hosts is coming soon.
"},{"location":"docs/guides/protips/#service-quotas","title":"Service quotas","text":"If you're using your own AWS, GCP, Azure, or OCI accounts, before you can use GPUs or spot instances, you have to request the corresponding service quotas for each type of instance in each region.
AWSCheck this guide on EC2 service quotas. The relevant service quotas include:
Running On-Demand P instances (on-demand V100, A100 80GB x8)All P4, P3 and P2 Spot Instance Requests (spot V100, A100 80GB x8)Running On-Demand G and VT instances (on-demand T4, A10G, L4)All G and VT Spot Instance Requests (spot T4, A10G, L4)Running Dedicated p5 Hosts (on-demand H100)All P5 Spot Instance Requests (spot H100)Check this guide on Compute Engine service quotas. The relevant service quotas include:
NVIDIA V100 GPUs (on-demand V100)Preemtible V100 GPUs (spot V100)NVIDIA T4 GPUs (on-demand T4)Preemtible T4 GPUs (spot T4)NVIDIA L4 GPUs (on-demand L4)Preemtible L4 GPUs (spot L4)NVIDIA A100 GPUs (on-demand A100)Preemtible A100 GPUs (spot A100)NVIDIA A100 80GB GPUs (on-demand A100 80GB)Preemtible A100 80GB GPUs (spot A100 80GB)NVIDIA H100 GPUs (on-demand H100)Preemtible H100 GPUs (spot H100)Check this guide on Azure service quotas. The relevant service quotas include:
Total Regional Spot vCPUs (any spot instances)Standard NCASv3_T4 Family vCPUs (on-demand T4)Standard NVADSA10v5 Family vCPUs (on-demand A10)Standard NCADS_A100_v4 Family vCPUs (on-demand A100 80GB)Standard NDASv4_A100 Family vCPUs (on-demand A100 40GB x8)Standard NDAMSv4_A100Family vCPUs (on-demand A100 80GB x8)Standard NCadsH100v5 Family vCPUs (on-demand H100)Standard NDSH100v5 Family vCPUs (on-demand H100 x8)Check this guide on requesting OCI service limits increase. The relevant service category is compute. The relevant resources include:
GPUs for GPU.A10 based VM and BM instances (on-demand A10)GPUs for GPU2 based VM and BM instances (on-demand P100)GPUs for GPU3 based VM and BM instances (on-demand V100)Note, for AWS, GCP, and Azure, service quota values are measured with the number of CPUs rather than GPUs.
"},{"location":"docs/installation/","title":"Installation","text":"To use the open-source version of dstack with your own cloud accounts or on-prem servers, you have to set up the server.
Follow the steps below to set up the server.
"},{"location":"docs/installation/#1-configure-backends","title":"1. Configure backends","text":"If you want the dstack server to run containers or manage clusters in your cloud accounts (or use Kubernetes), create the ~/.dstack/server/config.yml file and configure backends.
Once the ~/.dstack/server/config.yml file is configured, proceed to start the server:
$ pip install \"dstack[all]\" -U\n$ dstack server\n\nApplying ~/.dstack/server/config.yml...\n\nThe admin token is \"bbae0f28-d3dd-4820-bf61-8f4bb40815da\"\nThe server is running at http://127.0.0.1:3000/\n$ docker run -p 3000:3000 \\\n -v $HOME/.dstack/server/:/root/.dstack/server \\\n dstackai/dstack\n\nApplying ~/.dstack/server/config.yml...\n\nThe admin token is \"bbae0f28-d3dd-4820-bf61-8f4bb40815da\"\nThe server is running at http://127.0.0.1:3000/\nFor more details on how to deploy dstack using Docker, check its Docker repo.
By default, the dstack server stores its state in ~/.dstack/server/data using SQLite. To use a database, set the DSTACK_DATABASE_URL environment variable.
The server can be set up anywhere: on your laptop, a dedicated server, or in the cloud. Once the dstack server is up, you can use the CLI or API.
To point the CLI to the dstack server, configure it with the server address, user token and project name:
$ pip install dstack\n$ dstack config --url http://127.0.0.1:3000 \\\n --project main \\\n --token bbae0f28-d3dd-4820-bf61-8f4bb40815da\n\nConfiguration is updated at ~/.dstack/config.yml\nThis configuration is stored in ~/.dstack/config.yml.
Fleets
If you want the dstack server to run containers on your on-prem servers, use fleets.
If you don't want to host the dstack server yourself or would like to access GPU from the dstack marketplace, sign up with dstack Sky .
If you've signed up, open your project settings, and copy the dstack config command to point the CLI to the project.
Then, install the CLI on your machine and use the copied command.
$ pip install dstack\n$ dstack config --url https://sky.dstack.ai \\\n --project peterschmidt85 \\\n --token bbae0f28-d3dd-4820-bf61-8f4bb40815da\n\nConfiguration is updated at ~/.dstack/config.yml\nBy default, dstack Sky uses the GPU from its marketplace, which requires a credit card to be attached in your account settings.
To use your own cloud accounts, click the settings icon of the corresponding backend and specify credentials:
"},{"location":"docs/installation/#whats-next","title":"What's next?","text":"dev-environmenttaskserviceSometimes, you may want to reuse the same parameters across different .dstack.yml configurations.
This can be achieved by defining those parameters in a profile.
Profiles can be defined on the repository level (via the .dstack/profiles.yml file in the root directory of the repository) or on the global level (via the ~/.dstack/profiles.yml file).
Any profile can be marked as default so that it will be applied automatically for any run. Otherwise, you can refer to a specific profile via --profile NAME in dstack run.
profiles:\n - name: my-profile\n\n # The spot pololicy can be \"spot\", \"on-demand\", or \"auto\"\n spot_policy: auto\n\n # Limit the maximum price of the instance per hour\n max_price: 1.5\n\n # Stop any run if it runs longer that this duration\n max_duration: 1d\n\n # Use only these backends\n backends: [azure, lambda]\n\n # If set to true, this profile will be applied automatically\n default: true\nThe profile configuration supports many properties. See below.
"},{"location":"docs/reference/profiles.yml/#root-reference","title":"Root reference","text":""},{"location":"docs/reference/profiles.yml/#backends","title":"backends - (Optional) The backends to consider for provisioning (e.g., [aws, gcp]).","text":""},{"location":"docs/reference/profiles.yml/#regions","title":"regions - (Optional) The regions to consider for provisioning (e.g., [eu-west-1, us-west4, westeurope]).","text":""},{"location":"docs/reference/profiles.yml/#instance_types","title":"instance_types - (Optional) The cloud-specific instance types to consider for provisioning (e.g., [p3.8xlarge, n1-standard-4]).","text":""},{"location":"docs/reference/profiles.yml/#spot_policy","title":"spot_policy - (Optional) The policy for provisioning spot or on-demand instances: spot, on-demand, or auto.","text":""},{"location":"docs/reference/profiles.yml/#_retry","title":"retry - (Optional) The policy for resubmitting the run. Defaults to false.","text":""},{"location":"docs/reference/profiles.yml/#_retry_policy","title":"retry_policy - (Optional) The policy for resubmitting the run. Deprecated in favor of retry.","text":""},{"location":"docs/reference/profiles.yml/#max_duration","title":"max_duration - (Optional) The maximum duration of a run (e.g., 2h, 1d, etc). After it elapses, the run is forced to stop. Defaults to off.","text":""},{"location":"docs/reference/profiles.yml/#max_price","title":"max_price - (Optional) The maximum instance price per hour, in dollars.","text":""},{"location":"docs/reference/profiles.yml/#pool_name","title":"pool_name - (Optional) The name of the pool. If not set, dstack will use the default name.","text":""},{"location":"docs/reference/profiles.yml/#instance_name","title":"instance_name - (Optional) The name of the instance.","text":""},{"location":"docs/reference/profiles.yml/#creation_policy","title":"creation_policy - (Optional) The policy for using instances from the pool. Defaults to reuse-or-create.","text":""},{"location":"docs/reference/profiles.yml/#termination_policy","title":"termination_policy - (Optional) The policy for instance termination. Defaults to destroy-after-idle.","text":""},{"location":"docs/reference/profiles.yml/#termination_idle_time","title":"termination_idle_time - (Optional) Time to wait before destroying the idle instance. Defaults to 5m for dstack run and to 3d for dstack pool add.","text":""},{"location":"docs/reference/profiles.yml/#name","title":"name - The name of the profile that can be passed as --profile to dstack run.","text":""},{"location":"docs/reference/profiles.yml/#default","title":"default - (Optional) If set to true, dstack run will use this profile by default..","text":""},{"location":"docs/reference/profiles.yml/#retry","title":"retry","text":""},{"location":"docs/reference/profiles.yml/#on_events","title":"on_events - The list of events that should be handled with retry. Supported events are no-capacity, interruption, and error.","text":""},{"location":"docs/reference/profiles.yml/#duration","title":"duration - (Optional) The maximum period of retrying the run, e.g., 4h or 1d.","text":""},{"location":"docs/reference/api/python/","title":"Python API","text":"The Python API enables running tasks, services, and managing runs programmatically.
"},{"location":"docs/reference/api/python/#usage-example","title":"Usage example","text":"Below is a quick example of submitting a task for running and displaying its logs.
import sys\n\nfrom dstack.api import Task, GPU, Client, Resources\n\nclient = Client.from_config()\n\ntask = Task(\n image=\"ghcr.io/huggingface/text-generation-inference:latest\",\n env={\"MODEL_ID\": \"TheBloke/Llama-2-13B-chat-GPTQ\"},\n commands=[\n \"text-generation-launcher --trust-remote-code --quantize gptq\",\n ],\n ports=[\"80\"],\n resources=Resources(gpu=GPU(memory=\"24GB\")),\n)\n\nrun = client.runs.submit(\n run_name=\"my-awesome-run\", # If not specified, a random name is assigned \n configuration=task,\n repo=None, # Specify to mount additional files\n)\n\nrun.attach()\n\ntry:\n for log in run.logs():\n sys.stdout.buffer.write(log)\n sys.stdout.buffer.flush()\nexcept KeyboardInterrupt:\n run.stop(abort=True)\nfinally:\n run.detach()\nNOTE:
configuration argument in the submit method can be either dstack.api.Task or dstack.api.Service. dstack.api.Task or dstack.api.Service, you may specify the image argument. If image isn't specified, the default image will be used. For a private Docker registry, ensure you also pass the registry_auth argument.repo argument in the submit method allows the mounting of a local folder, a remote repo, or a programmatically created repo. In this case, the commands argument can refer to the files within this repo.attach method waits for the run to start and, for dstack.api.Task sets up an SSH tunnel and forwards configured ports to localhost.dstack.api","text":""},{"location":"docs/reference/api/python/#dstack.api.Client","title":"dstack.api.Client","text":"High-level API client for interacting with dstack server
Attributes:
Name Type Descriptionruns RunCollection Operations with runs.
repos RepoCollection Operations with repositories.
backends BackendCollection Operations with backends.
"},{"location":"docs/reference/api/python/#dstack.api.Client.from_config","title":"from_config(project_name=None, server_url=None, user_token=None, ssh_identity_file=None) staticmethod","text":"Creates a Client using the default configuration from ~/.dstack/config.yml if it exists.
Parameters:
Name Type Description Defaultproject_name Optional[str] The name of the project, required if server_url and user_token are specified
None server_url Optional[str] The dstack server URL (e.g. http://localhost:3000/ or https://sky.dstack.ai)
None user_token Optional[str] The dstack user token
None ssh_identity_file Optional[PathLike] The private SSH key path for SSH tunneling
None Returns:
Type DescriptionClient A client instance
"},{"location":"docs/reference/api/python/#dstack.api.Client.runs","title":"dstack.api.RunCollection","text":"Operations with runs
"},{"location":"docs/reference/api/python/#dstack.api.RunCollection.get","title":"get(run_name)","text":"Get run by run name
Parameters:
Name Type Description Defaultrun_name str run name
requiredReturns:
Type DescriptionOptional[Run] The run or None if not found
list(all=False)","text":"List runs
Parameters:
Name Type Description Defaultall bool show all runs (active and finished) if True
False Returns:
Type DescriptionList[Run] list of runs
"},{"location":"docs/reference/api/python/#dstack.api.RunCollection.submit","title":"submit(configuration, configuration_path=None, repo=None, backends=None, regions=None, instance_types=None, resources=None, spot_policy=None, retry_policy=None, max_duration=None, max_price=None, working_dir=None, run_name=None, reserve_ports=True)","text":"Submit a run
Parameters:
Name Type Description Defaultconfiguration Union[Task, Service] A run configuration.
requiredconfiguration_path Optional[str] The path to the configuration file, relative to the root directory of the repo.
None repo Union[LocalRepo, RemoteRepo, VirtualRepo] A repo to mount to the run.
None backends Optional[List[BackendType]] A list of allowed backend for provisioning.
None regions Optional[List[str]] A list of cloud regions for provisioning.
None resources Optional[ResourcesSpec] The requirements to run the configuration. Overrides the configuration's resources.
None spot_policy Optional[SpotPolicy] A spot policy for provisioning.
None retry_policy RetryPolicy A retry policy.
None max_duration Optional[Union[int, str]] The max instance running duration in seconds.
None max_price Optional[float] The max instance price in dollars per hour for provisioning.
None working_dir Optional[str] A working directory relative to the repo root directory
None run_name Optional[str] A desired name of the run. Must be unique in the project. If not specified, a random name is assigned.
None reserve_ports bool Whether local ports should be reserved in advance.
True Returns:
Type DescriptionRun submitted run
"},{"location":"docs/reference/api/python/#dstack.api.Client.repos","title":"dstack.api.RepoCollection","text":"Operations with repos
"},{"location":"docs/reference/api/python/#dstack.api.RepoCollection.init","title":"init(repo, git_identity_file=None, oauth_token=None)","text":"Initializes the repo and configures its credentials in the project. Must be invoked before mounting the repo to a run.
Example:
repo=RemoteRepo.from_url(\n repo_url=\"https://github.com/dstackai/dstack-examples\",\n repo_branch=\"main\",\n)\nclient.repos.init(repo)\nBy default, it uses the default Git credentials configured on the machine. You can override these credentials via the git_identity_file or oauth_token arguments of the init method.
Once the repo is initialized, you can pass the repo object to the run:
run = client.runs.submit(\n configuration=...,\n repo=repo,\n)\nParameters:
Name Type Description Defaultrepo Repo The repo to initialize.
requiredgit_identity_file Optional[PathLike] The private SSH key path for accessing the remote repo.
None oauth_token Optional[str] The GitHub OAuth token to access the remote repo.
None"},{"location":"docs/reference/api/python/#dstack.api.Task","title":"dstack.api.Task","text":""},{"location":"docs/reference/api/python/#nodes","title":"nodes - (Optional) Number of nodes. Defaults to 1.","text":""},{"location":"docs/reference/api/python/#name","title":"name - (Optional) The run name.","text":""},{"location":"docs/reference/api/python/#image","title":"image - (Optional) The name of the Docker image to run.","text":""},{"location":"docs/reference/api/python/#entrypoint","title":"entrypoint - (Optional) The Docker entrypoint.","text":""},{"location":"docs/reference/api/python/#working_dir","title":"working_dir - (Optional) The path to the working directory inside the container. It's specified relative to the repository directory (/workflow) and should be inside it. Defaults to \".\" .","text":""},{"location":"docs/reference/api/python/#home_dir","title":"home_dir - (Optional) The absolute path to the home directory inside the container. Defaults to /root. Defaults to /root.","text":""},{"location":"docs/reference/api/python/#_registry_auth","title":"registry_auth - (Optional) Credentials for pulling a private Docker image.","text":""},{"location":"docs/reference/api/python/#python","title":"python - (Optional) The major version of Python. Mutually exclusive with image.","text":""},{"location":"docs/reference/api/python/#env","title":"env - (Optional) The mapping or the list of environment variables.","text":""},{"location":"docs/reference/api/python/#setup","title":"setup - (Optional) The bash commands to run on the boot.","text":""},{"location":"docs/reference/api/python/#_resources","title":"resources - (Optional) The resources requirements to run the configuration.","text":""},{"location":"docs/reference/api/python/#_volumes","title":"volumes - (Optional) The volumes mount points.","text":""},{"location":"docs/reference/api/python/#ports","title":"ports - (Optional) Port numbers/mapping to expose.","text":""},{"location":"docs/reference/api/python/#commands","title":"commands - (Optional) The bash commands to run.","text":""},{"location":"docs/reference/api/python/#backends","title":"backends - (Optional) The backends to consider for provisioning (e.g., [aws, gcp]).","text":""},{"location":"docs/reference/api/python/#regions","title":"regions - (Optional) The regions to consider for provisioning (e.g., [eu-west-1, us-west4, westeurope]).","text":""},{"location":"docs/reference/api/python/#instance_types","title":"instance_types - (Optional) The cloud-specific instance types to consider for provisioning (e.g., [p3.8xlarge, n1-standard-4]).","text":""},{"location":"docs/reference/api/python/#spot_policy","title":"spot_policy - (Optional) The policy for provisioning spot or on-demand instances: spot, on-demand, or auto.","text":""},{"location":"docs/reference/api/python/#_retry","title":"retry - (Optional) The policy for resubmitting the run. Defaults to false.","text":""},{"location":"docs/reference/api/python/#_retry_policy","title":"retry_policy - (Optional) The policy for resubmitting the run. Deprecated in favor of retry.","text":""},{"location":"docs/reference/api/python/#max_duration","title":"max_duration - (Optional) The maximum duration of a run (e.g., 2h, 1d, etc). After it elapses, the run is forced to stop. Defaults to off.","text":""},{"location":"docs/reference/api/python/#max_price","title":"max_price - (Optional) The maximum instance price per hour, in dollars.","text":""},{"location":"docs/reference/api/python/#pool_name","title":"pool_name - (Optional) The name of the pool. If not set, dstack will use the default name.","text":""},{"location":"docs/reference/api/python/#instance_name","title":"instance_name - (Optional) The name of the instance.","text":""},{"location":"docs/reference/api/python/#creation_policy","title":"creation_policy - (Optional) The policy for using instances from the pool. Defaults to reuse-or-create.","text":""},{"location":"docs/reference/api/python/#termination_policy","title":"termination_policy - (Optional) The policy for instance termination. Defaults to destroy-after-idle.","text":""},{"location":"docs/reference/api/python/#termination_idle_time","title":"termination_idle_time - (Optional) Time to wait before destroying the idle instance. Defaults to 5m for dstack run and to 3d for dstack pool add.","text":""},{"location":"docs/reference/api/python/#dstack.api.Service","title":"dstack.api.Service","text":""},{"location":"docs/reference/api/python/#port","title":"port - The port, that application listens on or the mapping.","text":""},{"location":"docs/reference/api/python/#model","title":"model - (Optional) Mapping of the model for the OpenAI-compatible endpoint.","text":""},{"location":"docs/reference/api/python/#https","title":"https - (Optional) Enable HTTPS. Defaults to True.","text":""},{"location":"docs/reference/api/python/#auth","title":"auth - (Optional) Enable the authorization. Defaults to True.","text":""},{"location":"docs/reference/api/python/#replicas","title":"replicas - (Optional) The number of replicas. Can be a number (e.g. 2) or a range (0..4 or 1..8). If it's a range, the scaling property is required. Defaults to 1.","text":""},{"location":"docs/reference/api/python/#_scaling","title":"scaling - (Optional) The auto-scaling rules. Required if replicas is set to a range.","text":""},{"location":"docs/reference/api/python/#name","title":"name - (Optional) The run name.","text":""},{"location":"docs/reference/api/python/#image","title":"image - (Optional) The name of the Docker image to run.","text":""},{"location":"docs/reference/api/python/#entrypoint","title":"entrypoint - (Optional) The Docker entrypoint.","text":""},{"location":"docs/reference/api/python/#working_dir","title":"working_dir - (Optional) The path to the working directory inside the container. It's specified relative to the repository directory (/workflow) and should be inside it. Defaults to \".\" .","text":""},{"location":"docs/reference/api/python/#home_dir","title":"home_dir - (Optional) The absolute path to the home directory inside the container. Defaults to /root. Defaults to /root.","text":""},{"location":"docs/reference/api/python/#_registry_auth","title":"registry_auth - (Optional) Credentials for pulling a private Docker image.","text":""},{"location":"docs/reference/api/python/#python","title":"python - (Optional) The major version of Python. Mutually exclusive with image.","text":""},{"location":"docs/reference/api/python/#env","title":"env - (Optional) The mapping or the list of environment variables.","text":""},{"location":"docs/reference/api/python/#setup","title":"setup - (Optional) The bash commands to run on the boot.","text":""},{"location":"docs/reference/api/python/#_resources","title":"resources - (Optional) The resources requirements to run the configuration.","text":""},{"location":"docs/reference/api/python/#_volumes","title":"volumes - (Optional) The volumes mount points.","text":""},{"location":"docs/reference/api/python/#commands","title":"commands - (Optional) The bash commands to run.","text":""},{"location":"docs/reference/api/python/#backends","title":"backends - (Optional) The backends to consider for provisioning (e.g., [aws, gcp]).","text":""},{"location":"docs/reference/api/python/#regions","title":"regions - (Optional) The regions to consider for provisioning (e.g., [eu-west-1, us-west4, westeurope]).","text":""},{"location":"docs/reference/api/python/#instance_types","title":"instance_types - (Optional) The cloud-specific instance types to consider for provisioning (e.g., [p3.8xlarge, n1-standard-4]).","text":""},{"location":"docs/reference/api/python/#spot_policy","title":"spot_policy - (Optional) The policy for provisioning spot or on-demand instances: spot, on-demand, or auto.","text":""},{"location":"docs/reference/api/python/#_retry","title":"retry - (Optional) The policy for resubmitting the run. Defaults to false.","text":""},{"location":"docs/reference/api/python/#_retry_policy","title":"retry_policy - (Optional) The policy for resubmitting the run. Deprecated in favor of retry.","text":""},{"location":"docs/reference/api/python/#max_duration","title":"max_duration - (Optional) The maximum duration of a run (e.g., 2h, 1d, etc). After it elapses, the run is forced to stop. Defaults to off.","text":""},{"location":"docs/reference/api/python/#max_price","title":"max_price - (Optional) The maximum instance price per hour, in dollars.","text":""},{"location":"docs/reference/api/python/#pool_name","title":"pool_name - (Optional) The name of the pool. If not set, dstack will use the default name.","text":""},{"location":"docs/reference/api/python/#instance_name","title":"instance_name - (Optional) The name of the instance.","text":""},{"location":"docs/reference/api/python/#creation_policy","title":"creation_policy - (Optional) The policy for using instances from the pool. Defaults to reuse-or-create.","text":""},{"location":"docs/reference/api/python/#termination_policy","title":"termination_policy - (Optional) The policy for instance termination. Defaults to destroy-after-idle.","text":""},{"location":"docs/reference/api/python/#termination_idle_time","title":"termination_idle_time - (Optional) Time to wait before destroying the idle instance. Defaults to 5m for dstack run and to 3d for dstack pool add.","text":""},{"location":"docs/reference/api/python/#dstack.api.Run","title":"dstack.api.Run","text":"Attributes:
Name Type Descriptionname str run name
ports Optional[Dict[int, int]] ports mapping, if run is attached
backend Optional[BackendType] backend type
status RunStatus run status
hostname str instance hostname
"},{"location":"docs/reference/api/python/#dstack.api.Run.attach","title":"attach(ssh_identity_file=None)","text":"Establish an SSH tunnel to the instance and update SSH config
Parameters:
Name Type Description Defaultssh_identity_file Optional[PathLike] SSH keypair to access instances
None Raises:
Type DescriptionPortUsedError If ports are in use or the run is attached by another process.
"},{"location":"docs/reference/api/python/#dstack.api.Run.detach","title":"detach()","text":"Stop the SSH tunnel to the instance and update SSH config
"},{"location":"docs/reference/api/python/#dstack.api.Run.logs","title":"logs(start_time=None, diagnose=False, replica_num=0, job_num=0)","text":"Iterate through run's log messages
Parameters:
Name Type Description Defaultstart_time Optional[datetime] minimal log timestamp
None diagnose bool return runner logs if True
False Yields:
Type DescriptionIterable[bytes] log messages
"},{"location":"docs/reference/api/python/#dstack.api.Run.refresh","title":"refresh()","text":"Get up-to-date run info
"},{"location":"docs/reference/api/python/#dstack.api.Run.stop","title":"stop(abort=False)","text":"Terminate the instance and detach
Parameters:
Name Type Description Defaultabort bool gracefully stop the run if False
False"},{"location":"docs/reference/api/python/#dstack.api.Resources","title":"dstack.api.Resources","text":""},{"location":"docs/reference/api/python/#cpu","title":"cpu - (Optional) The number of CPU cores. Defaults to 2...","text":""},{"location":"docs/reference/api/python/#memory","title":"memory - (Optional) The RAM size (e.g., 8GB). Defaults to 8GB...","text":""},{"location":"docs/reference/api/python/#shm_size","title":"shm_size - (Optional) The size of shared memory (e.g., 8GB). If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure this.","text":""},{"location":"docs/reference/api/python/#_gpu","title":"gpu - (Optional) The GPU requirements.","text":""},{"location":"docs/reference/api/python/#_disk","title":"disk - (Optional) The disk resources.","text":""},{"location":"docs/reference/api/python/#dstack.api.GPU","title":"dstack.api.GPU","text":""},{"location":"docs/reference/api/python/#name","title":"name - (Optional) The name of the GPU (e.g., A100 or H100).","text":""},{"location":"docs/reference/api/python/#count","title":"count - (Optional) The number of GPUs. Defaults to 1.","text":""},{"location":"docs/reference/api/python/#memory","title":"memory - (Optional) The RAM size (e.g., 16GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).","text":""},{"location":"docs/reference/api/python/#total_memory","title":"total_memory - (Optional) The total RAM size (e.g., 32GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).","text":""},{"location":"docs/reference/api/python/#compute_capability","title":"compute_capability - (Optional) The minimum compute capability of the GPU (e.g., 7.5).","text":""},{"location":"docs/reference/api/python/#dstack.api.Disk","title":"dstack.api.Disk","text":""},{"location":"docs/reference/api/python/#size","title":"size - Disk size.","text":""},{"location":"docs/reference/api/python/#dstack.api.LocalRepo","title":"dstack.api.LocalRepo","text":"Creates an instance of a local repo from a local path.
Example:
run = client.runs.submit(\n configuration=...,\n repo=LocalRepo.from_dir(\".\"), # Mount the current folder to the run\n)\nfrom_dir(repo_dir) staticmethod","text":"Creates an instance of a local repo from a local path.
Parameters:
Name Type Description Defaultrepo_dir PathLike The path to a local folder
requiredReturns:
Type DescriptionLocalRepo A local repo instance
"},{"location":"docs/reference/api/python/#dstack.api.RemoteRepo","title":"dstack.api.RemoteRepo","text":"Creates an instance of a remote Git repo for mounting to a submitted run.
Using a locally checked-out remote Git repo:
repo=RemoteRepo.from_dir(repo_dir=\".\")\nUsing a remote Git repo by a URL:
repo=RemoteRepo.from_url(\n repo_url=\"https://github.com/dstackai/dstack-examples\",\n repo_branch=\"main\"\n)\nInitialize the repo before mounting it.
client.repos.init(repo)\nBy default, it uses the default Git credentials configured on the machine. You can override these credentials via the git_identity_file or oauth_token arguments of the init method.
Finally, you can pass the repo object to the run:
run = client.runs.submit(\n configuration=...,\n repo=repo,\n)\nfrom_dir(repo_dir) staticmethod","text":"Creates an instance of a remote repo from a local path.
Parameters:
Name Type Description Defaultrepo_dir PathLike The path to a local folder
requiredReturns:
Type DescriptionRemoteRepo A remote repo instance
"},{"location":"docs/reference/api/python/#dstack.api.RemoteRepo.from_url","title":"from_url(repo_url, repo_branch=None, repo_hash=None) staticmethod","text":"Creates an instance of a remote repo from a URL.
Parameters:
Name Type Description Defaultrepo_url str The URL of a remote Git repo
requiredrepo_branch Optional[str] The name of the remote branch. Must be specified if hash is not specified.
None repo_hash Optional[str] The hash of the revision. Must be specified if branch is not specified.
None Returns:
Type DescriptionRemoteRepo A remote repo instance
"},{"location":"docs/reference/api/python/#dstack.api.VirtualRepo","title":"dstack.api.VirtualRepo","text":"Allows mounting a repo created programmatically.
Example:
virtual_repo = VirtualRepo(repo_id=\"some-unique-repo-id\")\nvirtual_repo.add_file_from_package(package=some_package, path=\"requirements.txt\")\nvirtual_repo.add_file_from_package(package=some_package, path=\"train.py\")\n\nrun = client.runs.submit(\n configuration=...,\n repo=virtual_repo,\n)\nAttributes:
Name Type Descriptionrepo_id A unique identifier of the repo
"},{"location":"docs/reference/api/python/#dstack.api.VirtualRepo.add_file","title":"add_file(path, content)","text":"Adds a given file to the repo.
Attributes:
Name Type Descriptionpath str The path inside the repo to add the file.
content bytes The contents of the file.
"},{"location":"docs/reference/api/python/#dstack.api.VirtualRepo.add_file_from_package","title":"add_file_from_package(package, path)","text":"Includes a file from a given package to the repo.
Attributes:
Name Type Descriptionpackage Union[ModuleType, str] A package to include the file from.
path str The path to the file to include to the repo. Must be relative to the package directory.
"},{"location":"docs/reference/api/python/#dstack.api.RegistryAuth","title":"dstack.api.RegistryAuth","text":""},{"location":"docs/reference/api/python/#username","title":"username - The username.","text":""},{"location":"docs/reference/api/python/#password","title":"password - The password or access token.","text":""},{"location":"docs/reference/api/python/#dstack.api.Scaling","title":"dstack.api.Scaling","text":""},{"location":"docs/reference/api/python/#metric","title":"metric - The target metric to track. Currently, the only supported value is rps (meaning requests per second).","text":""},{"location":"docs/reference/api/python/#target","title":"target - The target value of the metric. The number of replicas is calculated based on this number and automatically adjusts (scales up or down) as this metric changes.","text":""},{"location":"docs/reference/api/python/#scale_up_delay","title":"scale_up_delay - (Optional) The delay in seconds before scaling up. Defaults to 300.","text":""},{"location":"docs/reference/api/python/#scale_down_delay","title":"scale_down_delay - (Optional) The delay in seconds before scaling down. Defaults to 600.","text":""},{"location":"docs/reference/api/python/#dstack.api.BackendType","title":"dstack.api.BackendType","text":"Attributes:
Name Type DescriptionAWS BackendType Amazon Web Services
AZURE BackendType Microsoft Azure
CUDO BackendType Cudo
DSTACK BackendType dstack Sky
GCP BackendType Google Cloud Platform
DATACRUNCH BackendType DataCrunch
KUBERNETES BackendType Kubernetes
LAMBDA BackendType Lambda Cloud
RUNPOD BackendType Runpod Cloud
TENSORDOCK BackendType TensorDock Marketplace
VASTAI BackendType Vast.ai Marketplace
"},{"location":"docs/reference/api/rest/","title":"REST API","text":""},{"location":"docs/reference/cli/","title":"CLI","text":""},{"location":"docs/reference/cli/#commands","title":"Commands","text":""},{"location":"docs/reference/cli/#dstack-server","title":"dstack server","text":"This command starts the dstack server.
$ dstack server --help\nUsage: dstack server [-h] [--host HOST] [-p PORT] [-l LOG_LEVEL] [--default]\n [--no-default] [--token TOKEN]\n\nOptions:\n -h, --help Show this help message and exit\n --host HOST Bind socket to this host. Defaults to 127.0.0.1\n -p, --port PORT Bind socket to this port. Defaults to 3000.\n -l, --log-level LOG_LEVEL\n Server logging level. Defaults to INFO.\n --default Update the default project configuration\n --no-default Do not update the default project configuration\n --token TOKEN The admin user token\nThis command must be called inside a folder before you can run dstack apply.
Git credentials
If the current folder is a remote Git repository, dstack init ensures that dstack can access it. By default, the command uses the remote repo's default Git credentials. These can be overridden with --git-identity (private SSH key) or --token (OAuth token).
$ dstack init --help\nUsage: dstack init [-h] [--project PROJECT] [-t OAUTH_TOKEN]\n [--git-identity SSH_PRIVATE_KEY]\n [--ssh-identity SSH_PRIVATE_KEY] [--local]\n\nOptions:\n -h, --help Show this help message and exit\n --project PROJECT The name of the project\n -t, --token OAUTH_TOKEN\n An authentication token for Git\n --git-identity SSH_PRIVATE_KEY\n The private SSH key path to access the remote repo\n --ssh-identity SSH_PRIVATE_KEY\n The private SSH key path for SSH tunneling\n --local Do not use git\nUser SSH key
By default, dstack uses its own SSH key to access instances (~/.dstack/ssh/id_rsa). It is possible to override this key via the --ssh-identity argument.
This command applies a given configuration. If a resource does not exist, dstack apply creates the resource. If a resource exists, dstack apply updates the resource in-place or re-creates the resource if the update is not possible.
$ dstack apply --help\nUsage: dstack apply [--project NAME] [-h [TYPE]] [-f FILE] [--force] [-y]\n\nOptions:\n --project NAME The name of the project. Defaults to $DSTACK_PROJECT\n -h, --help [TYPE] Show this help message and exit.\n -f, --file FILE The path to the configuration file. Defaults to\n $PWD/.dstack.yml\n --force Force apply when no changes detected\n -y, --yes Do not ask for confirmation\nThis command deletes the resources defined by a given configuration.
$ dstack delete --help\nUsage: dstack delete [-h] [--project NAME] [-f FILE] [-y]\n\nOptions:\n -h, --help Show this help message and exit\n --project NAME The name of the project. Defaults to $DSTACK_PROJECT\n -f, --file FILE The path to the configuration file. Defaults to\n $PWD/.dstack.yml\n -y, --yes Do not ask for confirmation\nNOTE:
The dstack delete command currently supports only gateway configurations. Support for other configuration types is coming soon.
This command shows the status of runs.
$ dstack ps --help\nUsage: dstack ps [-h] [--project NAME] [-a] [-v] [-w]\n\nOptions:\n -h, --help Show this help message and exit\n --project NAME The name of the project. Defaults to $DSTACK_PROJECT\n -a, --all Show all runs. By default, it only shows unfinished runs or\n the last finished.\n -v, --verbose Show more information about runs\n -w, --watch Watch statuses of runs in realtime\nThis command stops run(s) within the current repository.
$ dstack stop --help\nUsage: dstack stop [-h] [--project NAME] [-x] [-y] run_name\n\nPositional Arguments:\n run_name\n\nOptions:\n -h, --help Show this help message and exit\n --project NAME The name of the project. Defaults to $DSTACK_PROJECT\n -x, --abort\n -y, --yes\nThis command shows the output of a given run within the current repository.
$ dstack logs --help\nUsage: dstack logs [-h] [--project NAME] [-d] [-a]\n [--ssh-identity SSH_PRIVATE_KEY] [--replica REPLICA]\n [--job JOB]\n run_name\n\nPositional Arguments:\n run_name\n\nOptions:\n -h, --help Show this help message and exit\n --project NAME The name of the project. Defaults to $DSTACK_PROJECT\n -d, --diagnose\n -a, --attach Set up an SSH tunnel, and print logs as they follow.\n --ssh-identity SSH_PRIVATE_KEY\n The private SSH key path for SSH tunneling\n --replica REPLICA The relica number. Defaults to 0.\n --job JOB The job number inside the replica. Defaults to 0.\nBoth the CLI and API need to be configured with the server address, user token, and project name via ~/.dstack/config.yml.
At startup, the server automatically configures CLI and API with the server address, user token, and the default project name (main). This configuration is stored via ~/.dstack/config.yml.
To use CLI and API on different machines or projects, use the dstack config command.
$ dstack config --help\nUsage: dstack config [-h] [--project PROJECT] [--url URL] [--token TOKEN]\n [--default] [--remove] [--no-default]\n\nOptions:\n -h, --help Show this help message and exit\n --project PROJECT The name of the project to configure\n --url URL Server url\n --token TOKEN User token\n --default Set the project as default. It will be used when\n --project is omitted in commands.\n --remove Delete project configuration\n --no-default Do not prompt to set the project as default\nFleets enable efficient provisioning and management of clusters and instances.
"},{"location":"docs/reference/cli/#dstack-fleet-list","title":"dstack fleet list","text":"The dstack fleet list command displays fleets and instances.
$ dstack fleet list --help\nUsage: dstack fleet list [-h] [-w] [-v]\n\nOptions:\n -h, --help show this help message and exit\n -w, --watch Update listing in realtime\n -v, --verbose Show more information\nThe dstack fleet delete deletes fleets and instances. Cloud instances are terminated upon deletion.
$ dstack fleet delete --help\nUsage: dstack fleet delete [-h] [-i INSTANCE_NUM] [-y] name\n\nPositional Arguments:\n name The name of the fleet\n\nOptions:\n -h, --help show this help message and exit\n -i, --instance INSTANCE_NUM\n The instances to delete\n -y, --yes Don't ask for confirmation\nA gateway is required for running services. It handles ingress traffic, authorization, domain mapping, model mapping for the OpenAI-compatible endpoint, and so on.
"},{"location":"docs/reference/cli/#dstack-gateway-list","title":"dstack gateway list","text":"The dstack gateway list command displays the names and addresses of the gateways configured in the project.
$ dstack gateway list --help\nUsage: dstack gateway list [-h] [-w] [-v]\n\nOptions:\n -h, --help show this help message and exit\n -w, --watch Update listing in realtime\n -v, --verbose Show more information\nThe dstack gateway create command creates a new gateway instance in the project.
$ dstack gateway create --help\nUsage: dstack gateway create [-h] --backend {aws,azure,gcp,kubernetes}\n --region REGION [--set-default] [--name NAME]\n --domain DOMAIN\n\nOptions:\n -h, --help show this help message and exit\n --backend {aws,azure,gcp,kubernetes}\n --region REGION\n --set-default Set as default gateway for the project\n --name NAME Set a custom name for the gateway\n --domain DOMAIN Set the domain for the gateway\nThe dstack gateway delete command deletes the specified gateway.
$ dstack gateway delete --help\nUsage: dstack gateway delete [-h] [-y] name\n\nPositional Arguments:\n name The name of the gateway\n\nOptions:\n -h, --help show this help message and exit\n -y, --yes Don't ask for confirmation\nThe dstack gateway update command updates the specified gateway.
$ dstack gateway update --help\nUsage: dstack gateway update [-h] [--set-default] [--domain DOMAIN] name\n\nPositional Arguments:\n name The name of the gateway\n\nOptions:\n -h, --help show this help message and exit\n --set-default Set it the default gateway for the project\n --domain DOMAIN Set the domain for the gateway\nThe volumes commands.
"},{"location":"docs/reference/cli/#dstack-volume-list","title":"dstack volume list","text":"The dstack volume list command lists volumes.
$ dstack volume list --help\nUsage: dstack volume list [-h] [-w] [-v]\n\nOptions:\n -h, --help show this help message and exit\n -w, --watch Update listing in realtime\n -v, --verbose Show more information\nThe dstack volume delete command deletes volumes.
$ dstack volume delete --help\nUsage: dstack volume delete [-h] [-y] name\n\nPositional Arguments:\n name The name of the volume\n\nOptions:\n -h, --help show this help message and exit\n -y, --yes Don't ask for confirmation\nThis command runs a given configuration.
Deprecation
dstack run is deprecated in favor of dstack apply.
$ dstack run . --help\nUsage: dstack run [--project NAME] [-h [TYPE]] [-f FILE] [-y] [-n RUN_NAME]\n [-d] [--max-offers MAX_OFFERS] [-e KEY=VALUE] [--gpu SPEC]\n [--disk RANGE] [--profile NAME] [--max-price PRICE]\n [--max-duration DURATION] [-b NAME] [-r NAME]\n [--instance-type NAME]\n [--pool POOL_NAME | --reuse | --dont-destroy | --idle-duration IDLE_DURATION | --instance NAME]\n [--spot | --on-demand | --spot-auto | --spot-policy POLICY]\n [--retry | --no-retry | --retry-duration DURATION]\n working_dir\n\nPositional Arguments:\n working_dir\n\nOptions:\n --project NAME The name of the project. Defaults to $DSTACK_PROJECT\n -h, --help [TYPE] Show this help message and exit. TYPE is one of task,\n dev-environment, service\n -f, --file FILE The path to the configuration file. Defaults to\n $PWD/.dstack.yml\n -y, --yes Do not ask for confirmation\n -n, --name RUN_NAME The name of the run. If not specified, a random name\n is assigned\n -d, --detach Do not poll logs and run status\n --max-offers MAX_OFFERS\n Number of offers to show in the run plan\n -e, --env KEY=VALUE Environment variables\n --gpu SPEC Request GPU for the run. The format is\n NAME:COUNT:MEMORY (all parts are optional)\n --disk RANGE Request the size range of disk for the run. Example\n --disk 100GB...\n\nProfile:\n --profile NAME The name of the profile. Defaults to $DSTACK_PROFILE\n --max-price PRICE The maximum price per hour, in dollars\n --max-duration DURATION\n The maximum duration of the run\n -b, --backend NAME The backends that will be tried for provisioning\n -r, --region NAME The regions that will be tried for provisioning\n --instance-type NAME The cloud-specific instance types that will be tried\n for provisioning\n\nPools:\n --pool POOL_NAME The name of the pool. If not set, the default pool\n will be used\n --reuse Reuse instance from pool\n --dont-destroy Do not destroy instance after the run is finished\n --idle-duration IDLE_DURATION\n Time to wait before destroying the idle instance\n --instance NAME Reuse instance from pool with name NAME\n\nSpot Policy:\n --spot Consider only spot instances\n --on-demand Consider only on-demand instances\n --spot-auto Consider both spot and on-demand instances\n --spot-policy POLICY One of spot, on-demand, auto\n\nRetry Policy:\n --retry\n --no-retry\n --retry-duration DURATION\nWhen running anything via CLI, dstack uses the exact version of code from your project directory.
If there are large files, consider creating a .gitignore file to exclude them for better performance.
Pools allow for managing the lifecycle of instances and reusing them across runs. The default pool is created automatically.
Deprecation
Pools are deprecated in favor of fleets and will be removed in 0.19.0.
"},{"location":"docs/reference/cli/#dstack-pool-add","title":"dstack pool add","text":"The dstack pool add command provisions a cloud instance and adds it to a pool. If no pool name is specified, the instance goes to the default pool.
$ dstack pool add --help\nUsage: dstack pool add [-h] [-y] [--profile NAME] [--max-price PRICE]\n [-b NAME] [-r NAME] [--instance-type NAME]\n [--pool POOL_NAME] [--reuse] [--dont-destroy]\n [--idle-duration IDLE_DURATION]\n [--spot | --on-demand | --spot-auto | --spot-policy POLICY]\n [--retry | --no-retry | --retry-duration DURATION]\n [--cpu SPEC] [--memory SIZE] [--shared-memory SIZE]\n [--gpu SPEC] [--disk SIZE]\n\nOptions:\n -h, --help show this help message and exit\n -y, --yes Don't ask for confirmation\n --pool POOL_NAME The name of the pool. If not set, the default pool\n will be used\n --reuse Reuse instance from pool\n --dont-destroy Do not destroy instance after the run is finished\n --idle-duration IDLE_DURATION\n Time to wait before destroying the idle instance\n\nProfile:\n --profile NAME The name of the profile. Defaults to $DSTACK_PROFILE\n --max-price PRICE The maximum price per hour, in dollars\n -b, --backend NAME The backends that will be tried for provisioning\n -r, --region NAME The regions that will be tried for provisioning\n --instance-type NAME The cloud-specific instance types that will be tried\n for provisioning\n\nSpot Policy:\n --spot Consider only spot instances\n --on-demand Consider only on-demand instances\n --spot-auto Consider both spot and on-demand instances\n --spot-policy POLICY One of spot, on-demand, auto\n\nRetry Policy:\n --retry\n --no-retry\n --retry-duration DURATION\n\nResources:\n --cpu SPEC Request the CPU count. Default: 2..\n --memory SIZE Request the size of RAM. The format is SIZE:MB|GB|TB.\n Default: 8GB..\n --shared-memory SIZE Request the size of Shared Memory. The format is\n SIZE:MB|GB|TB.\n --gpu SPEC Request GPU for the run. The format is\n NAME:COUNT:MEMORY (all parts are optional)\n --disk SIZE Request the size of disk for the run. Example --disk\n 100GB...\nThe dstack pool add-ssh command adds an existing remote instance to a pool. If no pool name is specified, the instance goes to the default pool.
$ dstack pool add-ssh --help\nUsage: dstack pool add-ssh [-h] -i SSH_PRIVATE_KEY [-p SSH_PORT]\n [-l LOGIN_NAME] [--region REGION]\n [--pool POOL_NAME] [--name INSTANCE_NAME]\n [--network NETWORK]\n destination\n\nPositional Arguments:\n destination\n\nOptions:\n -h, --help show this help message and exit\n -i SSH_PRIVATE_KEY The private SSH key path for SSH\n -p SSH_PORT SSH port to connect\n -l LOGIN_NAME User to login\n --region REGION Host region\n --pool POOL_NAME Pool name\n --name INSTANCE_NAME Set the name of the instance\n --network NETWORK Network address for multinode setup. Format <ip\n address>/<netmask>\nThe dstack pool ps command lists all active instances of a pool. If no pool name is specified, default pool instances are displayed.
$ dstack pool ps --help\nUsage: dstack pool ps [-h] [--pool POOL_NAME] [-w]\n\nShow instances in the pool\n\nOptions:\n -h, --help show this help message and exit\n --pool POOL_NAME The name of the pool. If not set, the default pool will be\n used\n -w, --watch Watch instances in realtime\nThe dstack pool rm command removes an instance from a pool. Cloud instances are terminated upon removal.
$ dstack pool rm --help\nUsage: dstack pool rm [-h] [--pool POOL_NAME] [--force] [-y] instance_name\n\nPositional Arguments:\n instance_name The name of the instance\n\nOptions:\n -h, --help show this help message and exit\n --pool POOL_NAME The name of the pool. If not set, the default pool will be\n used\n --force The name of the instance\n -y, --yes Don't ask for confirmation\nThe dstack pool create command creates a new pool.
$ dstack pool create --help\nUsage: dstack pool create [-h] -n POOL_NAME\n\nOptions:\n -h, --help show this help message and exit\n -n, --name POOL_NAME The name of the pool\nThe dstack pool list command lists all existing pools.
$ dstack pool list --help\nUsage: dstack pool list [-h] [-v VERBOSE]\n\nList available pools\n\nOptions:\n -h, --help show this help message and exit\n -v, --verbose VERBOSE\n Show more information\nThe dstack pool set-default command sets the project's default pool.
$ dstack pool set-default --help\nUsage: dstack pool set-default [-h] --pool POOL_NAME\n\nOptions:\n -h, --help show this help message and exit\n --pool POOL_NAME The name of the pool\nThe dstack pool delete command deletes a specified pool.
$ dstack pool delete --help\nUsage: dstack pool delete [-h] -n POOL_NAME\n\nOptions:\n -h, --help show this help message and exit\n -n, --name POOL_NAME The name of the pool\nDSTACK_CLI_LOG_LEVEL \u2013 (Optional) Configures CLI logging level. Defaults to INFO.DSTACK_SERVER_LOG_LEVEL \u2013 (Optional) Has the same effect as --log-level. Defaults to INFO.DSTACK_SERVER_HOST \u2013 (Optional) Has the same effect as --host. Defaults to 127.0.0.1.DSTACK_SERVER_PORT \u2013 (Optional) Has the same effect as --port. Defaults to 3000.DSTACK_SERVER_ADMIN_TOKEN \u2013 (Optional) Has the same effect as --token. Defaults to None.DSTACK_DATABASE_URL \u2013 (Optional) The database URL to use instead of default SQLite. Currently dstack supports Postgres. Example: postgresql+asyncpg://myuser:mypassword@localhost:5432/mydatabase. Defaults to None.DSTACK_SERVER_DIR \u2013 (Optional) Sets path to store data and server configs. Defaults to ~/.dstack/server.DSTACK_SERVER_ROOT_LOG_LEVEL \u2013 (Optional) Sets root logger log level. Defaults to ERROR.DSTACK_SERVER_LOG_FORMAT \u2013 (Optional) Sets format of log output. Can be rich, standard, json.. Defaults to rich.DSTACK_SERVER_UVICORN_LOG_LEVEL \u2013 (Optional) Sets uvicorn logger log level. Defaults to ERROR.DSTACK_PROFILE \u2013 (Optional) Has the same effect as --profile. Defaults to None.DSTACK_PROJECT \u2013 (Optional) Has the same effect as --project. Defaults to None.DSTACK_RUNNER_VERSION \u2013 (Optional) Sets exact runner version for debug. Defaults to latest.DSTACK_DEFAULT_CREDS_DISABLED \u2013 (Optional) Disables default credentials detection if set. Defaults to None.DSTACK_LOCAL_BACKEND_ENABLED \u2013 (Optional) Enables local backend for debug if set. Defaults to None.The dev-environment configuration type allows running dev environments.
Configuration files must have a name ending with .dstack.yml (e.g., .dstack.yml or serve.dstack.yml are both acceptable) and can be located in the project's root directory or any nested folder. Any configuration can be run via dstack run.
If you don't specify image, dstack uses the default Docker image pre-configured with python, pip, conda (Miniforge), and essential CUDA drivers. The python property determines which default Docker image is used.
type: dev-environment\n\npython: \"3.11\"\n\nide: vscode\nnvcc
Note that the default Docker image doesn't bundle nvcc, which is required for building custom CUDA kernels. To install it, use conda install cuda.
type: dev-environment\n\nimage: ghcr.io/huggingface/text-generation-inference:latest\n\nide: vscode\nUse the registry_auth property to provide credentials for a private Docker registry.
type: dev-environment\n\nimage: ghcr.io/huggingface/text-generation-inference:latest\nregistry_auth:\n username: peterschmidt85\n password: ghp_e49HcZ9oYwBzUbcSk2080gXZOU2hiT9AeSR5\n\nide: vscode\nIf you specify memory size, you can either specify an explicit size (e.g. 24GB) or a range (e.g. 24GB.., or 24GB..80GB, or ..80GB).
type: dev-environment\n\nide: vscode\n\nresources:\n # 200GB or more RAM\n memory: 200GB..\n\n # 4 GPUs from 40GB to 80GB\n gpu: 40GB..80GB:4\n\n # Shared memory\n shm_size: 16GB\n\n disk: 500GB\nThe gpu property allows specifying not only memory size but also GPU names and their quantity. Examples: A100 (one A100), A10G,A100 (either A10G or A100), A100:80GB (one A100 of 80GB), A100:2 (two A100), 24GB..40GB:2 (two GPUs between 24GB and 40GB), A100:40GB:2 (two A100 GPUs of 40GB).
To use TPUs, specify its architecture prefixed by tpu- via the gpu property.
type: dev-environment\n\nide: vscode\n\nresources:\n gpu: tpu-v2-8\nCurrently, only 8 TPU cores can be specified, supporting single TPU device workloads. Multi-TPU support is coming soon.
Shared memoryIf you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure shm_size, e.g. set it to 16GB.
type: dev-environment\n\nenv:\n - HUGGING_FACE_HUB_TOKEN\n - HF_HUB_ENABLE_HF_TRANSFER=1\n\nide: vscode\nIf you don't assign a value to an environment variable (see HUGGING_FACE_HUB_TOKEN above), dstack will require the value to be passed via the CLI or set in the current process.
For instance, you can define environment variables in a .env file and utilize tools like direnv.
The following environment variables are available in any run and are passed by dstack by default:
DSTACK_RUN_NAME The name of the run DSTACK_REPO_ID The ID of the repo DSTACK_GPUS_NUM The total number of GPUs in the run"},{"location":"docs/reference/dstack.yml/dev-environment/#spot-policy","title":"Spot policy","text":"You can choose whether to use spot instances, on-demand instances, or any available type.
type: dev-environment\n\nide: vscode\n\nspot_policy: auto\nThe spot_policy accepts spot, on-demand, and auto. The default for dev environments is on-demand.
By default, dstack provisions instances in all configured backends. However, you can specify the list of backends:
type: dev-environment\n\nide: vscode\n\nbackends: [aws, gcp]\nBy default, dstack uses all configured regions. However, you can specify the list of regions:
type: dev-environment\n\nide: vscode\n\nregions: [eu-west-1, eu-west-2]\nVolumes allow you to persist data between runs. To attach a volume, simply specify its name using the volumes property and specify where to mount its contents:
type: dev-environment\n\nide: vscode\n\nvolumes:\n - name: my-new-volume\n path: /volume_data\nOnce you run this configuration, the contents of the volume will be attached to /volume_data inside the dev environment, and its contents will persist across runs.
Limitations
When you're running a dev environment, task, or service with dstack, it automatically mounts the project folder contents to /workflow (and sets that as the current working directory). Right now, dstack doesn't allow you to attach volumes to /workflow or any of its subdirectories.
The dev-environment configuration type supports many other options. See below.
ide - The IDE to run.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#version","title":"version - (Optional) The version of the IDE.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#init","title":"init - (Optional) The bash commands to run.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#name","title":"name - (Optional) The run name.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#image","title":"image - (Optional) The name of the Docker image to run.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#entrypoint","title":"entrypoint - (Optional) The Docker entrypoint.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#working_dir","title":"working_dir - (Optional) The path to the working directory inside the container. It's specified relative to the repository directory (/workflow) and should be inside it. Defaults to \".\" .","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#home_dir","title":"home_dir - (Optional) The absolute path to the home directory inside the container. Defaults to /root. Defaults to /root.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#_registry_auth","title":"registry_auth - (Optional) Credentials for pulling a private Docker image.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#python","title":"python - (Optional) The major version of Python. Mutually exclusive with image.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#env","title":"env - (Optional) The mapping or the list of environment variables.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#setup","title":"setup - (Optional) The bash commands to run on the boot.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#_resources","title":"resources - (Optional) The resources requirements to run the configuration.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#_volumes","title":"volumes - (Optional) The volumes mount points.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#ports","title":"ports - (Optional) Port numbers/mapping to expose.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#backends","title":"backends - (Optional) The backends to consider for provisioning (e.g., [aws, gcp]).","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#regions","title":"regions - (Optional) The regions to consider for provisioning (e.g., [eu-west-1, us-west4, westeurope]).","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#instance_types","title":"instance_types - (Optional) The cloud-specific instance types to consider for provisioning (e.g., [p3.8xlarge, n1-standard-4]).","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#spot_policy","title":"spot_policy - (Optional) The policy for provisioning spot or on-demand instances: spot, on-demand, or auto.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#_retry","title":"retry - (Optional) The policy for resubmitting the run. Defaults to false.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#_retry_policy","title":"retry_policy - (Optional) The policy for resubmitting the run. Deprecated in favor of retry.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#max_duration","title":"max_duration - (Optional) The maximum duration of a run (e.g., 2h, 1d, etc). After it elapses, the run is forced to stop. Defaults to off.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#max_price","title":"max_price - (Optional) The maximum instance price per hour, in dollars.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#pool_name","title":"pool_name - (Optional) The name of the pool. If not set, dstack will use the default name.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#instance_name","title":"instance_name - (Optional) The name of the instance.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#creation_policy","title":"creation_policy - (Optional) The policy for using instances from the pool. Defaults to reuse-or-create.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#termination_policy","title":"termination_policy - (Optional) The policy for instance termination. Defaults to destroy-after-idle.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#termination_idle_time","title":"termination_idle_time - (Optional) Time to wait before destroying the idle instance. Defaults to 5m for dstack run and to 3d for dstack pool add.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#resources","title":"resources","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#cpu","title":"cpu - (Optional) The number of CPU cores. Defaults to 2...","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#memory","title":"memory - (Optional) The RAM size (e.g., 8GB). Defaults to 8GB...","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#shm_size","title":"shm_size - (Optional) The size of shared memory (e.g., 8GB). If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure this.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#_gpu","title":"gpu - (Optional) The GPU requirements. Can be set to a number, a string (e.g. A100, 80GB:2, etc.), or an object.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#_disk","title":"disk - (Optional) The disk resources.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#resources-gpu","title":"resources.gpu","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#name","title":"name - (Optional) The GPU name or list of names.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#count","title":"count - (Optional) The number of GPUs. Defaults to 1.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#memory","title":"memory - (Optional) The RAM size (e.g., 16GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#total_memory","title":"total_memory - (Optional) The total RAM size (e.g., 32GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#compute_capability","title":"compute_capability - (Optional) The minimum compute capability of the GPU (e.g., 7.5).","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#resources-disk","title":"resources.disk","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#size","title":"size - The disk size. Can be a string (e.g., 100GB or 100GB..) or an object.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#registry_auth","title":"registry_auth","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#username","title":"username - The username.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#password","title":"password - The password or access token.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#volumes_1","title":"volumes","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#name","title":"name - The name of the volume to mount.","text":""},{"location":"docs/reference/dstack.yml/dev-environment/#path","title":"path - The container path to mount the volume at.","text":""},{"location":"docs/reference/dstack.yml/fleet/","title":"fleet","text":"The fleet configuration type allows creating and updating fleets.
Configuration files must have a name ending with .dstack.yml (e.g., .dstack.yml or fleet.dstack.yml are both acceptable) and can be located in the project's root directory or any nested folder. Any configuration can be applied via dstack apply.
type: fleet\nname: my-gcp-fleet\nnodes: 4\nplacement: cluster\nbackends: [gcp]\nresources:\n gpu: 1\ntype: fleet\nname: my-ssh-fleet\nssh_config:\n user: ubuntu\n identity_file: ~/.ssh/id_rsa\n hosts:\n - 3.255.177.51\n - 3.255.177.52\nname - (Optional) The fleet name.","text":""},{"location":"docs/reference/dstack.yml/fleet/#_ssh_config","title":"ssh_config - (Optional) The parameters for adding instances via SSH.","text":""},{"location":"docs/reference/dstack.yml/fleet/#nodes","title":"nodes - (Optional) The number of instances.","text":""},{"location":"docs/reference/dstack.yml/fleet/#placement","title":"placement - (Optional) The placement of instances: any or cluster.","text":""},{"location":"docs/reference/dstack.yml/fleet/#_resources","title":"resources - (Optional) The resources requirements.","text":""},{"location":"docs/reference/dstack.yml/fleet/#backends","title":"backends - (Optional) The backends to consider for provisioning (e.g., [aws, gcp]).","text":""},{"location":"docs/reference/dstack.yml/fleet/#regions","title":"regions - (Optional) The regions to consider for provisioning (e.g., [eu-west-1, us-west4, westeurope]).","text":""},{"location":"docs/reference/dstack.yml/fleet/#instance_types","title":"instance_types - (Optional) The cloud-specific instance types to consider for provisioning (e.g., [p3.8xlarge, n1-standard-4]).","text":""},{"location":"docs/reference/dstack.yml/fleet/#spot_policy","title":"spot_policy - (Optional) The policy for provisioning spot or on-demand instances: spot, on-demand, or auto.","text":""},{"location":"docs/reference/dstack.yml/fleet/#_retry","title":"retry - (Optional) The policy for provisioning retry. Defaults to false.","text":""},{"location":"docs/reference/dstack.yml/fleet/#max_price","title":"max_price - (Optional) The maximum instance price per hour, in dollars.","text":""},{"location":"docs/reference/dstack.yml/fleet/#termination_policy","title":"termination_policy - (Optional) The policy for instance termination. Defaults to destroy-after-idle.","text":""},{"location":"docs/reference/dstack.yml/fleet/#termination_idle_time","title":"termination_idle_time - (Optional) Time to wait before destroying idle instances. Defaults to 3d.","text":""},{"location":"docs/reference/dstack.yml/fleet/#ssh","title":"ssh","text":""},{"location":"docs/reference/dstack.yml/fleet/#user","title":"user - (Optional) The user to log in with on all hosts.","text":""},{"location":"docs/reference/dstack.yml/fleet/#port","title":"port - (Optional) The SSH port to connect to.","text":""},{"location":"docs/reference/dstack.yml/fleet/#identity_file","title":"identity_file - (Optional) The private key to use for all hosts.","text":""},{"location":"docs/reference/dstack.yml/fleet/#hosts","title":"hosts - The per host connection parameters: a hostname or an object that overrides default ssh parameters.","text":""},{"location":"docs/reference/dstack.yml/fleet/#network","title":"network - (Optional) The network address for cluster setup in the format <ip>/<netmask>.","text":""},{"location":"docs/reference/dstack.yml/fleet/#sshhostsn","title":"ssh.hosts[n]","text":""},{"location":"docs/reference/dstack.yml/fleet/#hostname","title":"hostname - The IP address or domain to connect to.","text":""},{"location":"docs/reference/dstack.yml/fleet/#port","title":"port - (Optional) The SSH port to connect to for this host.","text":""},{"location":"docs/reference/dstack.yml/fleet/#user","title":"user - (Optional) The user to log in with for this host.","text":""},{"location":"docs/reference/dstack.yml/fleet/#identity_file","title":"identity_file - (Optional) The private key to use for this host.","text":""},{"location":"docs/reference/dstack.yml/fleet/#resources","title":"resources","text":""},{"location":"docs/reference/dstack.yml/fleet/#cpu","title":"cpu - (Optional) The number of CPU cores. Defaults to 2...","text":""},{"location":"docs/reference/dstack.yml/fleet/#memory","title":"memory - (Optional) The RAM size (e.g., 8GB). Defaults to 8GB...","text":""},{"location":"docs/reference/dstack.yml/fleet/#shm_size","title":"shm_size - (Optional) The size of shared memory (e.g., 8GB). If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure this.","text":""},{"location":"docs/reference/dstack.yml/fleet/#_gpu","title":"gpu - (Optional) The GPU requirements. Can be set to a number, a string (e.g. A100, 80GB:2, etc.), or an object.","text":""},{"location":"docs/reference/dstack.yml/fleet/#_disk","title":"disk - (Optional) The disk resources.","text":""},{"location":"docs/reference/dstack.yml/fleet/#resources-gpu","title":"resouces.gpu","text":""},{"location":"docs/reference/dstack.yml/fleet/#name","title":"name - (Optional) The GPU name or list of names.","text":""},{"location":"docs/reference/dstack.yml/fleet/#count","title":"count - (Optional) The number of GPUs. Defaults to 1.","text":""},{"location":"docs/reference/dstack.yml/fleet/#memory","title":"memory - (Optional) The RAM size (e.g., 16GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).","text":""},{"location":"docs/reference/dstack.yml/fleet/#total_memory","title":"total_memory - (Optional) The total RAM size (e.g., 32GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).","text":""},{"location":"docs/reference/dstack.yml/fleet/#compute_capability","title":"compute_capability - (Optional) The minimum compute capability of the GPU (e.g., 7.5).","text":""},{"location":"docs/reference/dstack.yml/fleet/#resources-disk","title":"resouces.disk","text":""},{"location":"docs/reference/dstack.yml/fleet/#size","title":"size - The disk size. Can be a string (e.g., 100GB or 100GB..) or an object.","text":""},{"location":"docs/reference/dstack.yml/fleet/#retry","title":"retry","text":""},{"location":"docs/reference/dstack.yml/fleet/#on_events","title":"on_events - The list of events that should be handled with retry. Supported events are no-capacity, interruption, and error.","text":""},{"location":"docs/reference/dstack.yml/fleet/#duration","title":"duration - (Optional) The maximum period of retrying the run, e.g., 4h or 1d.","text":""},{"location":"docs/reference/dstack.yml/gateway/","title":"gateway","text":"The gateway configuration type allows creating and updating gateways.
Configuration files must have a name ending with .dstack.yml (e.g., .dstack.yml or gateway.dstack.yml are both acceptable) and can be located in the project's root directory or any nested folder. Any configuration can be applied via dstack apply.
type: gateway\nname: example-gateway\n\nbackend: aws\nregion: eu-west-1\ndomain: example.com\nname - (Optional) The gateway name.","text":""},{"location":"docs/reference/dstack.yml/gateway/#default","title":"default - (Optional) Make the gateway default.","text":""},{"location":"docs/reference/dstack.yml/gateway/#backend","title":"backend - The gateway backend.","text":""},{"location":"docs/reference/dstack.yml/gateway/#region","title":"region - The gateway region.","text":""},{"location":"docs/reference/dstack.yml/gateway/#domain","title":"domain - (Optional) The gateway domain, e.g. example.com.","text":""},{"location":"docs/reference/dstack.yml/gateway/#public_ip","title":"public_ip - (Optional) Allocate public IP for the gateway. Defaults to True.","text":""},{"location":"docs/reference/dstack.yml/gateway/#_certificate","title":"certificate - (Optional) The SSL certificate configuration. Defaults to type: lets-encrypt.","text":""},{"location":"docs/reference/dstack.yml/gateway/#certificatetypelets-encrypt","title":"certificate[type=lets-encrypt]","text":""},{"location":"docs/reference/dstack.yml/gateway/#type","title":"type - Automatic certificates by Let's Encrypt. Must be lets-encrypt.","text":""},{"location":"docs/reference/dstack.yml/gateway/#certificatetypeacm","title":"certificate[type=acm]","text":""},{"location":"docs/reference/dstack.yml/gateway/#type","title":"type - Certificates by AWS Certificate Manager (ACM). Must be acm.","text":""},{"location":"docs/reference/dstack.yml/gateway/#arn","title":"arn - The ARN of the wildcard ACM certificate for the domain.","text":""},{"location":"docs/reference/dstack.yml/service/","title":"service","text":"The service configuration type allows running services.
Configuration files must have a name ending with .dstack.yml (e.g., .dstack.yml or serve.dstack.yml are both acceptable) and can be located in the project's root directory or any nested folder. Any configuration can be run via dstack run . -f PATH.
If you don't specify image, dstack uses the default Docker image pre-configured with python, pip, conda (Miniforge), and essential CUDA drivers. The python property determines which default Docker image is used.
type: service\n\npython: \"3.11\"\n\ncommands:\n - python3 -m http.server\n\nport: 8000\nnvcc
Note that the default Docker image doesn't bundle nvcc, which is required for building custom CUDA kernels. To install it, use conda install cuda.
type: service\n\nimage: dstackai/base:py3.11-0.4-cuda-12.1\n\ncommands:\n - python3 -m http.server\n\nport: 8000\nUse the registry_auth property to provide credentials for a private Docker registry.
type: service\n\nimage: dstackai/base:py3.11-0.4-cuda-12.1\n\ncommands:\n - python3 -m http.server\nregistry_auth:\n username: peterschmidt85\n password: ghp_e49HcZ9oYwBzUbcSk2080gXZOU2hiT9AeSR5\n\nport: 8000\nBy default, if you run a service, its endpoint is accessible at https://<run name>.<gateway domain>.
If you run a model, you can optionally configure the mapping to make it accessible via the OpenAI-compatible interface.
type: service\n\npython: \"3.11\"\n\nenv:\n - MODEL=NousResearch/Llama-2-7b-chat-hf\ncommands:\n - pip install vllm\n - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000\nport: 8000\n\nresources:\n gpu: 24GB\n\n# Enable the OpenAI-compatible endpoint\nmodel:\n format: openai\n type: chat\n name: NousResearch/Llama-2-7b-chat-hf\nIn this case, with such a configuration, once the service is up, you'll be able to access the model at https://gateway.<gateway domain> via the OpenAI-compatible interface.
The format supports only tgi (Text Generation Inference) and openai (if you are using Text Generation Inference or vLLM with OpenAI-compatible mode).
By default, dstack loads the chat template from the model's repository. If it is not present there, manual configuration is required.
type: service\n\nimage: ghcr.io/huggingface/text-generation-inference:latest\nenv:\n - MODEL_ID=TheBloke/Llama-2-13B-chat-GPTQ\ncommands:\n - text-generation-launcher --port 8000 --trust-remote-code --quantize gptq\nport: 8000\n\nresources:\n gpu: 80GB\n\n# Enable the OpenAI-compatible endpoint\nmodel:\n type: chat\n name: TheBloke/Llama-2-13B-chat-GPTQ\n format: tgi\n chat_template: \"{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% set system_message = false %}{% endif %}{% for message in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if loop.index0 == 0 and system_message != false %}{% set content = '<<SYS>>\\\\n' + system_message + '\\\\n<</SYS>>\\\\n\\\\n' + message['content'] %}{% else %}{% set content = message['content'] %}{% endif %}{% if message['role'] == 'user' %}{{ '<s>[INST] ' + content.strip() + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ ' ' + content.strip() + ' </s>' }}{% endif %}{% endfor %}\"\n eos_token: \"</s>\"\nPlease note that model mapping is an experimental feature with the following limitations:
chat_template uses bos_token. As a workaround, replace bos_token inside chat_template with the token content itself.eos_token is defined in the model repository as a dictionary. As a workaround, set eos_token manually, as shown in the example above (see Chat template).If you encounter any other issues, please make sure to file a GitHub issue.
"},{"location":"docs/reference/dstack.yml/service/#auto-scaling","title":"Auto-scaling","text":"By default, dstack runs a single replica of the service. You can configure the number of replicas as well as the auto-scaling rules.
type: service\n\npython: \"3.11\"\n\nenv:\n - MODEL=NousResearch/Llama-2-7b-chat-hf\ncommands:\n - pip install vllm\n - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000\nport: 8000\n\nresources:\n gpu: 24GB\n\n# Enable the OpenAI-compatible endpoint\nmodel:\n format: openai\n type: chat\n name: NousResearch/Llama-2-7b-chat-hf\n\nreplicas: 1..4\nscaling:\n metric: rps\n target: 10\nThe replicas property can be a number or a range.
The metric property of scaling only supports the rps metric (requests per second). In this case dstack adjusts the number of replicas (scales up or down) automatically based on the load.
Setting the minimum number of replicas to 0 allows the service to scale down to zero when there are no requests.
If you specify memory size, you can either specify an explicit size (e.g. 24GB) or a range (e.g. 24GB.., or 24GB..80GB, or ..80GB).
type: service\n\npython: \"3.11\"\ncommands:\n - pip install vllm\n - python -m vllm.entrypoints.openai.api_server\n --model mistralai/Mixtral-8X7B-Instruct-v0.1\n --host 0.0.0.0\n --tensor-parallel-size 2 # Match the number of GPUs\nport: 8000\n\nresources:\n # 2 GPUs of 80GB\n gpu: 80GB:2\n\n disk: 200GB\n\n# Enable the OpenAI-compatible endpoint\nmodel:\n type: chat\n name: TheBloke/Mixtral-8x7B-Instruct-v0.1-GPTQ\n format: openai\nThe gpu property allows specifying not only memory size but also GPU names and their quantity. Examples: A100 (one A100), A10G,A100 (either A10G or A100), A100:80GB (one A100 of 80GB), A100:2 (two A100), 24GB..40GB:2 (two GPUs between 24GB and 40GB), A100:40GB:2 (two A100 GPUs of 40GB).
If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure shm_size, e.g. set it to 16GB.
By default, the service endpoint requires the Authorization header with \"Bearer <dstack token>\". Authorization can be disabled by setting auth to false.
type: service\n\npython: \"3.11\"\n\ncommands:\n - python3 -m http.server\n\nport: 8000\n\nauth: false\ntype: service\n\npython: \"3.11\"\n\nenv:\n - HUGGING_FACE_HUB_TOKEN\n - MODEL=NousResearch/Llama-2-7b-chat-hf\ncommands:\n - pip install vllm\n - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000\nport: 8000\n\nresources:\n gpu: 24GB\nIf you don't assign a value to an environment variable (see HUGGING_FACE_HUB_TOKEN above), dstack will require the value to be passed via the CLI or set in the current process.
For instance, you can define environment variables in a .env file and utilize tools like direnv.
The following environment variables are available in any run and are passed by dstack by default:
DSTACK_RUN_NAME The name of the run DSTACK_REPO_ID The ID of the repo DSTACK_GPUS_NUM The total number of GPUs in the run"},{"location":"docs/reference/dstack.yml/service/#spot-policy","title":"Spot policy","text":"You can choose whether to use spot instances, on-demand instances, or any available type.
type: service\n\ncommands:\n - python3 -m http.server\n\nport: 8000\n\nspot_policy: auto\nThe spot_policy accepts spot, on-demand, and auto. The default for services is auto.
By default, dstack provisions instances in all configured backends. However, you can specify the list of backends:
type: service\n\ncommands:\n - python3 -m http.server\n\nport: 8000\n\nbackends: [aws, gcp]\nBy default, dstack uses all configured regions. However, you can specify the list of regions:
type: service\n\ncommands:\n - python3 -m http.server\n\nport: 8000\n\nregions: [eu-west-1, eu-west-2]\nVolumes allow you to persist data between runs. To attach a volume, simply specify its name using the volumes property and specify where to mount its contents:
type: service\n\ncommands:\n - python3 -m http.server\n\nport: 8000\n\nvolumes:\n - name: my-new-volume\n path: /volume_data\nOnce you run this configuration, the contents of the volume will be attached to /volume_data inside the service, and its contents will persist across runs.
The service configuration type supports many other options. See below.
port - The port, that application listens on or the mapping.","text":""},{"location":"docs/reference/dstack.yml/service/#model","title":"model - (Optional) Mapping of the model for the OpenAI-compatible endpoint.","text":""},{"location":"docs/reference/dstack.yml/service/#https","title":"https - (Optional) Enable HTTPS. Defaults to True.","text":""},{"location":"docs/reference/dstack.yml/service/#auth","title":"auth - (Optional) Enable the authorization. Defaults to True.","text":""},{"location":"docs/reference/dstack.yml/service/#replicas","title":"replicas - (Optional) The number of replicas. Can be a number (e.g. 2) or a range (0..4 or 1..8). If it's a range, the scaling property is required. Defaults to 1.","text":""},{"location":"docs/reference/dstack.yml/service/#_scaling","title":"scaling - (Optional) The auto-scaling rules. Required if replicas is set to a range.","text":""},{"location":"docs/reference/dstack.yml/service/#name","title":"name - (Optional) The run name.","text":""},{"location":"docs/reference/dstack.yml/service/#image","title":"image - (Optional) The name of the Docker image to run.","text":""},{"location":"docs/reference/dstack.yml/service/#entrypoint","title":"entrypoint - (Optional) The Docker entrypoint.","text":""},{"location":"docs/reference/dstack.yml/service/#working_dir","title":"working_dir - (Optional) The path to the working directory inside the container. It's specified relative to the repository directory (/workflow) and should be inside it. Defaults to \".\" .","text":""},{"location":"docs/reference/dstack.yml/service/#home_dir","title":"home_dir - (Optional) The absolute path to the home directory inside the container. Defaults to /root. Defaults to /root.","text":""},{"location":"docs/reference/dstack.yml/service/#_registry_auth","title":"registry_auth - (Optional) Credentials for pulling a private Docker image.","text":""},{"location":"docs/reference/dstack.yml/service/#python","title":"python - (Optional) The major version of Python. Mutually exclusive with image.","text":""},{"location":"docs/reference/dstack.yml/service/#env","title":"env - (Optional) The mapping or the list of environment variables.","text":""},{"location":"docs/reference/dstack.yml/service/#setup","title":"setup - (Optional) The bash commands to run on the boot.","text":""},{"location":"docs/reference/dstack.yml/service/#_resources","title":"resources - (Optional) The resources requirements to run the configuration.","text":""},{"location":"docs/reference/dstack.yml/service/#_volumes","title":"volumes - (Optional) The volumes mount points.","text":""},{"location":"docs/reference/dstack.yml/service/#commands","title":"commands - (Optional) The bash commands to run.","text":""},{"location":"docs/reference/dstack.yml/service/#backends","title":"backends - (Optional) The backends to consider for provisioning (e.g., [aws, gcp]).","text":""},{"location":"docs/reference/dstack.yml/service/#regions","title":"regions - (Optional) The regions to consider for provisioning (e.g., [eu-west-1, us-west4, westeurope]).","text":""},{"location":"docs/reference/dstack.yml/service/#instance_types","title":"instance_types - (Optional) The cloud-specific instance types to consider for provisioning (e.g., [p3.8xlarge, n1-standard-4]).","text":""},{"location":"docs/reference/dstack.yml/service/#spot_policy","title":"spot_policy - (Optional) The policy for provisioning spot or on-demand instances: spot, on-demand, or auto.","text":""},{"location":"docs/reference/dstack.yml/service/#_retry","title":"retry - (Optional) The policy for resubmitting the run. Defaults to false.","text":""},{"location":"docs/reference/dstack.yml/service/#_retry_policy","title":"retry_policy - (Optional) The policy for resubmitting the run. Deprecated in favor of retry.","text":""},{"location":"docs/reference/dstack.yml/service/#max_duration","title":"max_duration - (Optional) The maximum duration of a run (e.g., 2h, 1d, etc). After it elapses, the run is forced to stop. Defaults to off.","text":""},{"location":"docs/reference/dstack.yml/service/#max_price","title":"max_price - (Optional) The maximum instance price per hour, in dollars.","text":""},{"location":"docs/reference/dstack.yml/service/#pool_name","title":"pool_name - (Optional) The name of the pool. If not set, dstack will use the default name.","text":""},{"location":"docs/reference/dstack.yml/service/#instance_name","title":"instance_name - (Optional) The name of the instance.","text":""},{"location":"docs/reference/dstack.yml/service/#creation_policy","title":"creation_policy - (Optional) The policy for using instances from the pool. Defaults to reuse-or-create.","text":""},{"location":"docs/reference/dstack.yml/service/#termination_policy","title":"termination_policy - (Optional) The policy for instance termination. Defaults to destroy-after-idle.","text":""},{"location":"docs/reference/dstack.yml/service/#termination_idle_time","title":"termination_idle_time - (Optional) Time to wait before destroying the idle instance. Defaults to 5m for dstack run and to 3d for dstack pool add.","text":""},{"location":"docs/reference/dstack.yml/service/#model_1","title":"model","text":""},{"location":"docs/reference/dstack.yml/service/#type","title":"type - The type of the model.","text":""},{"location":"docs/reference/dstack.yml/service/#name","title":"name - The name of the model.","text":""},{"location":"docs/reference/dstack.yml/service/#format","title":"format - The serving format. Supported values include openai and tgi.","text":""},{"location":"docs/reference/dstack.yml/service/#scaling","title":"scaling","text":""},{"location":"docs/reference/dstack.yml/service/#metric","title":"metric - The target metric to track. Currently, the only supported value is rps (meaning requests per second).","text":""},{"location":"docs/reference/dstack.yml/service/#target","title":"target - The target value of the metric. The number of replicas is calculated based on this number and automatically adjusts (scales up or down) as this metric changes.","text":""},{"location":"docs/reference/dstack.yml/service/#scale_up_delay","title":"scale_up_delay - (Optional) The delay in seconds before scaling up. Defaults to 300.","text":""},{"location":"docs/reference/dstack.yml/service/#scale_down_delay","title":"scale_down_delay - (Optional) The delay in seconds before scaling down. Defaults to 600.","text":""},{"location":"docs/reference/dstack.yml/service/#resources","title":"resources","text":""},{"location":"docs/reference/dstack.yml/service/#cpu","title":"cpu - (Optional) The number of CPU cores. Defaults to 2...","text":""},{"location":"docs/reference/dstack.yml/service/#memory","title":"memory - (Optional) The RAM size (e.g., 8GB). Defaults to 8GB...","text":""},{"location":"docs/reference/dstack.yml/service/#shm_size","title":"shm_size - (Optional) The size of shared memory (e.g., 8GB). If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure this.","text":""},{"location":"docs/reference/dstack.yml/service/#_gpu","title":"gpu - (Optional) The GPU requirements. Can be set to a number, a string (e.g. A100, 80GB:2, etc.), or an object.","text":""},{"location":"docs/reference/dstack.yml/service/#_disk","title":"disk - (Optional) The disk resources.","text":""},{"location":"docs/reference/dstack.yml/service/#resources-gpu","title":"resouces.gpu","text":""},{"location":"docs/reference/dstack.yml/service/#name","title":"name - (Optional) The GPU name or list of names.","text":""},{"location":"docs/reference/dstack.yml/service/#count","title":"count - (Optional) The number of GPUs. Defaults to 1.","text":""},{"location":"docs/reference/dstack.yml/service/#memory","title":"memory - (Optional) The RAM size (e.g., 16GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).","text":""},{"location":"docs/reference/dstack.yml/service/#total_memory","title":"total_memory - (Optional) The total RAM size (e.g., 32GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).","text":""},{"location":"docs/reference/dstack.yml/service/#compute_capability","title":"compute_capability - (Optional) The minimum compute capability of the GPU (e.g., 7.5).","text":""},{"location":"docs/reference/dstack.yml/service/#resources-disk","title":"resouces.disk","text":""},{"location":"docs/reference/dstack.yml/service/#size","title":"size - The disk size. Can be a string (e.g., 100GB or 100GB..) or an object.","text":""},{"location":"docs/reference/dstack.yml/service/#registry_auth","title":"registry_auth","text":""},{"location":"docs/reference/dstack.yml/service/#username","title":"username - The username.","text":""},{"location":"docs/reference/dstack.yml/service/#password","title":"password - The password or access token.","text":""},{"location":"docs/reference/dstack.yml/service/#volumes_1","title":"volumes","text":""},{"location":"docs/reference/dstack.yml/service/#name","title":"name - The name of the volume to mount.","text":""},{"location":"docs/reference/dstack.yml/service/#path","title":"path - The container path to mount the volume at.","text":""},{"location":"docs/reference/dstack.yml/task/","title":"task","text":"The task configuration type allows running tasks.
Configuration files must have a name ending with .dstack.yml (e.g., .dstack.yml or serve.dstack.yml are both acceptable) and can be located in the project's root directory or any nested folder. Any configuration can be run via dstack run.
If you don't specify image, dstack uses the default Docker image pre-configured with python, pip, conda (Miniforge), and essential CUDA drivers. The python property determines which default Docker image is used.
type: task\n\npython: \"3.11\"\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py\nnvcc
Note that the default Docker image doesn't bundle nvcc, which is required for building custom CUDA kernels. To install it, use conda install cuda.
A task can configure ports. In this case, if the task is running an application on a port, dstack run will securely allow you to access this port from your local machine through port forwarding.
type: task\n\npython: \"3.11\"\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - tensorboard --logdir results/runs &\n - python fine-tuning/qlora/train.py\n\nports:\n - 6000\nWhen running it, dstack run forwards 6000 port to localhost:6000, enabling secure access.
type: dev-environment\n\nimage: dstackai/base:py3.11-0.4-cuda-12.1\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py\nUse the registry_auth property to provide credentials for a private Docker registry.
type: dev-environment\n\nimage: dstackai/base:py3.11-0.4-cuda-12.1\nregistry_auth:\n username: peterschmidt85\n password: ghp_e49HcZ9oYwBzUbcSk2080gXZOU2hiT9AeSR5\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py\nIf you specify memory size, you can either specify an explicit size (e.g. 24GB) or a range (e.g. 24GB.., or 24GB..80GB, or ..80GB).
type: task\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py\n\nresources:\n # 200GB or more RAM\n memory: 200GB..\n\n # 4 GPUs from 40GB to 80GB\n gpu: 40GB..80GB:4\n\n # Shared memory\n shm_size: 16GB\n\n disk: 500GB\nThe gpu property allows specifying not only memory size but also GPU names and their quantity. Examples: A100 (one A100), A10G,A100 (either A10G or A100), A100:80GB (one A100 of 80GB), A100:2 (two A100), 24GB..40GB:2 (two GPUs between 24GB and 40GB), A100:40GB:2 (two A100 GPUs of 40GB).
To use TPUs, specify its architecture prefixed by tpu- via the gpu property.
type: task\n\npython: \"3.11\"\n\ncommands:\n - pip install torch~=2.3.0 torch_xla[tpu]~=2.3.0 torchvision -f https://storage.googleapis.com/libtpu-releases/index.html\n - git clone --recursive https://github.com/pytorch/xla.git\n - python3 xla/test/test_train_mp_imagenet.py --fake_data --model=resnet50 --num_epochs=1\n\nresources:\n gpu: tpu-v2-8\nCurrently, only 8 TPU cores can be specified, supporting single host workloads. Multi-host support is coming soon.
Shared memoryIf you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure shm_size, e.g. set it to 16GB.
type: task\n\npython: \"3.11\"\n\nenv:\n - HUGGING_FACE_HUB_TOKEN\n - HF_HUB_ENABLE_HF_TRANSFER=1\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py\nIf you don't assign a value to an environment variable (see HUGGING_FACE_HUB_TOKEN above), dstack will require the value to be passed via the CLI or set in the current process.
For instance, you can define environment variables in a .env file and utilize tools like direnv.
The following environment variables are available in any run and are passed by dstack by default:
DSTACK_RUN_NAME The name of the run DSTACK_REPO_ID The ID of the repo DSTACK_GPUS_NUM The total number of GPUs in the run DSTACK_NODES_NUM The number of nodes in the run DSTACK_NODE_RANK The rank of the node DSTACK_MASTER_NODE_IP The internal IP address the master node"},{"location":"docs/reference/dstack.yml/task/#_nodes","title":"Distributed tasks","text":"By default, the task runs on a single node. However, you can run it on a cluster of nodes.
type: task\n\n# The size of the cluster\nnodes: 2\n\npython: \"3.11\"\nenv:\n - HF_HUB_ENABLE_HF_TRANSFER=1\ncommands:\n - pip install -r requirements.txt\n - torchrun\n --nproc_per_node=$DSTACK_GPUS_PER_NODE\n --node_rank=$DSTACK_NODE_RANK\n --nnodes=$DSTACK_NODES_NUM\n --master_addr=$DSTACK_MASTER_NODE_IP\n --master_port=8008 resnet_ddp.py\n --num_epochs 20\n\nresources:\n gpu: 24GB\nIf you run the task, dstack first provisions the master node and then runs the other nodes of the cluster. All nodes are provisioned in the same region.
dstack is easy to use with accelerate, torchrun, and other distributed frameworks. All you need to do is pass the corresponding environment variables such as DSTACK_GPUS_PER_NODE, DSTACK_NODE_RANK, DSTACK_NODES_NUM, DSTACK_MASTER_NODE_IP, and DSTACK_GPUS_NUM (see System environment variables).
Running on multiple nodes is supported only with aws, gcp, azure, oci, and instances added via dstack pool add-ssh.
You can parameterize tasks with user arguments using ${{ run.args }} in the configuration.
type: task\n\npython: \"3.11\"\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py ${{ run.args }}\nNow, you can pass your arguments to the dstack run command:
$ dstack run . -f train.dstack.yml --train_batch_size=1 --num_train_epochs=100\nHere's an example of using ports to run web apps with tasks.
type: task\n\npython: \"3.11\"\n\ncommands:\n - pip3 install streamlit\n - streamlit hello\n\nports: \n - 8501\nYou can choose whether to use spot instances, on-demand instances, or any available type.
type: task\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py\n\nspot_policy: auto\nThe spot_policy accepts spot, on-demand, and auto. The default for tasks is auto.
By default, dstack provisions instances in all configured backends. However, you can specify the list of backends:
type: task\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py\n\nbackends: [aws, gcp]\nBy default, dstack uses all configured regions. However, you can specify the list of regions:
type: task\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py\n\nregions: [eu-west-1, eu-west-2]\nVolumes allow you to persist data between runs. To attach a volume, simply specify its name using the volumes property and specify where to mount its contents:
type: task\n\npython: \"3.11\"\n\ncommands:\n - pip install -r fine-tuning/qlora/requirements.txt\n - python fine-tuning/qlora/train.py\n\nvolumes:\n - name: my-new-volume\n path: /volume_data\nOnce you run this configuration, the contents of the volume will be attached to /volume_data inside the task, and its contents will persist across runs.
Limitations
When you're running a dev environment, task, or service with dstack, it automatically mounts the project folder contents to /workflow (and sets that as the current working directory). Right now, dstack doesn't allow you to attach volumes to /workflow or any of its subdirectories.
The task configuration type supports many other options. See below.
nodes - (Optional) Number of nodes. Defaults to 1.","text":""},{"location":"docs/reference/dstack.yml/task/#name","title":"name - (Optional) The run name.","text":""},{"location":"docs/reference/dstack.yml/task/#image","title":"image - (Optional) The name of the Docker image to run.","text":""},{"location":"docs/reference/dstack.yml/task/#entrypoint","title":"entrypoint - (Optional) The Docker entrypoint.","text":""},{"location":"docs/reference/dstack.yml/task/#working_dir","title":"working_dir - (Optional) The path to the working directory inside the container. It's specified relative to the repository directory (/workflow) and should be inside it. Defaults to \".\" .","text":""},{"location":"docs/reference/dstack.yml/task/#home_dir","title":"home_dir - (Optional) The absolute path to the home directory inside the container. Defaults to /root. Defaults to /root.","text":""},{"location":"docs/reference/dstack.yml/task/#_registry_auth","title":"registry_auth - (Optional) Credentials for pulling a private Docker image.","text":""},{"location":"docs/reference/dstack.yml/task/#python","title":"python - (Optional) The major version of Python. Mutually exclusive with image.","text":""},{"location":"docs/reference/dstack.yml/task/#env","title":"env - (Optional) The mapping or the list of environment variables.","text":""},{"location":"docs/reference/dstack.yml/task/#setup","title":"setup - (Optional) The bash commands to run on the boot.","text":""},{"location":"docs/reference/dstack.yml/task/#_resources","title":"resources - (Optional) The resources requirements to run the configuration.","text":""},{"location":"docs/reference/dstack.yml/task/#_volumes","title":"volumes - (Optional) The volumes mount points.","text":""},{"location":"docs/reference/dstack.yml/task/#ports","title":"ports - (Optional) Port numbers/mapping to expose.","text":""},{"location":"docs/reference/dstack.yml/task/#commands","title":"commands - (Optional) The bash commands to run.","text":""},{"location":"docs/reference/dstack.yml/task/#backends","title":"backends - (Optional) The backends to consider for provisioning (e.g., [aws, gcp]).","text":""},{"location":"docs/reference/dstack.yml/task/#regions","title":"regions - (Optional) The regions to consider for provisioning (e.g., [eu-west-1, us-west4, westeurope]).","text":""},{"location":"docs/reference/dstack.yml/task/#instance_types","title":"instance_types - (Optional) The cloud-specific instance types to consider for provisioning (e.g., [p3.8xlarge, n1-standard-4]).","text":""},{"location":"docs/reference/dstack.yml/task/#spot_policy","title":"spot_policy - (Optional) The policy for provisioning spot or on-demand instances: spot, on-demand, or auto.","text":""},{"location":"docs/reference/dstack.yml/task/#_retry","title":"retry - (Optional) The policy for resubmitting the run. Defaults to false.","text":""},{"location":"docs/reference/dstack.yml/task/#_retry_policy","title":"retry_policy - (Optional) The policy for resubmitting the run. Deprecated in favor of retry.","text":""},{"location":"docs/reference/dstack.yml/task/#max_duration","title":"max_duration - (Optional) The maximum duration of a run (e.g., 2h, 1d, etc). After it elapses, the run is forced to stop. Defaults to off.","text":""},{"location":"docs/reference/dstack.yml/task/#max_price","title":"max_price - (Optional) The maximum instance price per hour, in dollars.","text":""},{"location":"docs/reference/dstack.yml/task/#pool_name","title":"pool_name - (Optional) The name of the pool. If not set, dstack will use the default name.","text":""},{"location":"docs/reference/dstack.yml/task/#instance_name","title":"instance_name - (Optional) The name of the instance.","text":""},{"location":"docs/reference/dstack.yml/task/#creation_policy","title":"creation_policy - (Optional) The policy for using instances from the pool. Defaults to reuse-or-create.","text":""},{"location":"docs/reference/dstack.yml/task/#termination_policy","title":"termination_policy - (Optional) The policy for instance termination. Defaults to destroy-after-idle.","text":""},{"location":"docs/reference/dstack.yml/task/#termination_idle_time","title":"termination_idle_time - (Optional) Time to wait before destroying the idle instance. Defaults to 5m for dstack run and to 3d for dstack pool add.","text":""},{"location":"docs/reference/dstack.yml/task/#resources","title":"resources","text":""},{"location":"docs/reference/dstack.yml/task/#cpu","title":"cpu - (Optional) The number of CPU cores. Defaults to 2...","text":""},{"location":"docs/reference/dstack.yml/task/#memory","title":"memory - (Optional) The RAM size (e.g., 8GB). Defaults to 8GB...","text":""},{"location":"docs/reference/dstack.yml/task/#shm_size","title":"shm_size - (Optional) The size of shared memory (e.g., 8GB). If you are using parallel communicating processes (e.g., dataloaders in PyTorch), you may need to configure this.","text":""},{"location":"docs/reference/dstack.yml/task/#_gpu","title":"gpu - (Optional) The GPU requirements. Can be set to a number, a string (e.g. A100, 80GB:2, etc.), or an object.","text":""},{"location":"docs/reference/dstack.yml/task/#_disk","title":"disk - (Optional) The disk resources.","text":""},{"location":"docs/reference/dstack.yml/task/#resources-gpu","title":"resouces.gpu","text":""},{"location":"docs/reference/dstack.yml/task/#name","title":"name - (Optional) The GPU name or list of names.","text":""},{"location":"docs/reference/dstack.yml/task/#count","title":"count - (Optional) The number of GPUs. Defaults to 1.","text":""},{"location":"docs/reference/dstack.yml/task/#memory","title":"memory - (Optional) The RAM size (e.g., 16GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).","text":""},{"location":"docs/reference/dstack.yml/task/#total_memory","title":"total_memory - (Optional) The total RAM size (e.g., 32GB). Can be set to a range (e.g. 16GB.., or 16GB..80GB).","text":""},{"location":"docs/reference/dstack.yml/task/#compute_capability","title":"compute_capability - (Optional) The minimum compute capability of the GPU (e.g., 7.5).","text":""},{"location":"docs/reference/dstack.yml/task/#resources-disk","title":"resouces.disk","text":""},{"location":"docs/reference/dstack.yml/task/#size","title":"size - The disk size. Can be a string (e.g., 100GB or 100GB..) or an object.","text":""},{"location":"docs/reference/dstack.yml/task/#registry_auth","title":"registry_auth","text":""},{"location":"docs/reference/dstack.yml/task/#username","title":"username - The username.","text":""},{"location":"docs/reference/dstack.yml/task/#password","title":"password - The password or access token.","text":""},{"location":"docs/reference/dstack.yml/task/#volumesn","title":"volumes[n]","text":""},{"location":"docs/reference/dstack.yml/task/#name","title":"name - The name of the volume to mount.","text":""},{"location":"docs/reference/dstack.yml/task/#path","title":"path - The container path to mount the volume at.","text":""},{"location":"docs/reference/dstack.yml/volume/","title":"volume","text":"The volume configuration type allows creating, registering, and updating volumes.
Configuration files must have a name ending with .dstack.yml (e.g., .dstack.yml or vol.dstack.yml are both acceptable) and can be located in the project's root directory or any nested folder. Any configuration can be applied via dstack apply.
type: volume\nname: my-aws-volume\nbackend: aws\nregion: eu-central-1\nsize: 100GB\ntype: volume\nname: my-external-volume\nbackend: aws\nregion: eu-central-1\nvolume_id: vol1235\nname - (Optional) The volume name.","text":""},{"location":"docs/reference/dstack.yml/volume/#backend","title":"backend - The volume backend.","text":""},{"location":"docs/reference/dstack.yml/volume/#region","title":"region - The volume region.","text":""},{"location":"docs/reference/dstack.yml/volume/#size","title":"size - (Optional) The volume size. Must be specified when creating new volumes.","text":""},{"location":"docs/reference/dstack.yml/volume/#volume_id","title":"volume_id - (Optional) The volume ID. Must be specified when registering external volumes.","text":""},{"location":"docs/reference/server/config.yml/","title":"~/.dstack/server/config.yml","text":"The ~/.dstack/server/config.yml file is used by the dstack server to configure cloud accounts.
The dstack server allows you to configure backends for multiple projects. If you don't need multiple projects, use only the main project.
Each cloud account must be configured under the backends property of the respective project. See the examples below.
There are two ways to configure AWS: using an access key or using the default credentials.
Access keyDefault credentialsCreate an access key by following the this guide . Once you've downloaded the .csv file with your IAM user's Access key ID and Secret access key, proceed to configure the backend.
projects:\n- name: main\n backends:\n - type: aws\n creds:\n type: access_key\n access_key: KKAAUKLIZ5EHKICAOASV\n secret_key: pn158lMqSBJiySwpQ9ubwmI6VUU3/W2fdJdFwfgO\nIf you have default credentials set up (e.g. in ~/.aws/credentials), configure the backend like this:
projects:\n - name: main\n backends:\n - type: aws\n creds:\n type: default\nBy default, dstack uses the default VPC. It's possible to customize it:
projects:\n - name: main\n backends:\n - type: aws\n creds:\n type: default\n\n vpc_name: my-vpc\nprojects:\n - name: main\n backends:\n - type: aws\n creds:\n type: default\n\n default_vpcs: true\n vpc_ids:\n us-east-1: vpc-0a2b3c4d5e6f7g8h\n us-east-2: vpc-9i8h7g6f5e4d3c2b\n us-west-1: vpc-4d3c2b1a0f9e8d7\nFor the regions without configured vpc_ids, enable default VPCs by setting default_vpcs to true.
The following AWS policy permissions are sufficient for dstack to work:
{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"ec2:AttachVolume\",\n \"ec2:AuthorizeSecurityGroupEgress\",\n \"ec2:AuthorizeSecurityGroupIngress\",\n \"ec2:CancelSpotInstanceRequests\",\n \"ec2:CreateSecurityGroup\",\n \"ec2:CreateTags\",\n \"ec2:CreateVolume\",\n \"ec2:DeleteVolume\",\n \"ec2:DescribeAvailabilityZones\",\n \"ec2:DescribeImages\",\n \"ec2:DescribeInstances\",\n \"ec2:DescribeInstanceAttribute\",\n \"ec2:DescribeRouteTables\",\n \"ec2:DescribeSecurityGroups\",\n \"ec2:DescribeSubnets\",\n \"ec2:DescribeVpcs\",\n \"ec2:DescribeVolumes\",\n \"ec2:DetachVolume\",\n \"ec2:RunInstances\",\n \"ec2:TerminateInstances\"\n ],\n \"Resource\": \"*\"\n },\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"servicequotas:ListServiceQuotas\",\n \"servicequotas:GetServiceQuota\"\n ],\n \"Resource\": \"*\"\n },\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"elasticloadbalancing:CreateLoadBalancer\",\n \"elasticloadbalancing:CreateTargetGroup\",\n \"elasticloadbalancing:CreateListener\",\n \"elasticloadbalancing:RegisterTargets\",\n \"elasticloadbalancing:AddTags\",\n \"elasticloadbalancing:DeleteLoadBalancer\",\n \"elasticloadbalancing:DeleteTargetGroup\",\n \"elasticloadbalancing:DeleteListener\",\n \"elasticloadbalancing:DeregisterTargets\"\n ],\n \"Resource\": \"*\"\n },\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"acm:DescribeCertificate\",\n \"acm:ListCertificates\"\n ],\n \"Resource\": \"*\"\n }\n ]\n}\nThe elasticloadbalancing:* and acm:* permissions are only needed for provisioning gateways with ACM (AWS Certificate Manager) certificates.
By default, dstack utilizes public subnets and permits inbound SSH traffic exclusively for any provisioned instances. If you want dstack to use private subnets, set public_ips to false.
projects:\n - name: main\n backends:\n - type: aws\n creds:\n type: default\n\n public_ips: false\nUsing private subnets assumes that both the dstack server and users can access the configured VPC's private subnets (e.g., through VPC peering).
There are two ways to configure Azure: using a client secret or using the default credentials.
Client secretDefault credentialsA client secret can be created using the Azure CLI :
SUBSCRIPTION_ID=...\naz ad sp create-for-rbac\n --name dstack-app \\\n --role $DSTACK_ROLE \\\n --scopes /subscriptions/$SUBSCRIPTION_ID \\\n --query \"{ tenant_id: tenant, client_id: appId, client_secret: password }\"\nOnce you have tenant_id, client_id, and client_secret, go ahead and configure the backend.
projects:\n- name: main\n backends:\n - type: azure\n subscription_id: 06c82ce3-28ff-4285-a146-c5e981a9d808\n tenant_id: f84a7584-88e4-4fd2-8e97-623f0a715ee1\n creds:\n type: client\n client_id: acf3f73a-597b-46b6-98d9-748d75018ed0\n client_secret: 1Kb8Q~o3Q2hdEvrul9yaj5DJDFkuL3RG7lger2VQ\nObtain the subscription_id and tenant_id via the Azure CLI :
az account show --query \"{subscription_id: id, tenant_id: tenantId}\"\nThen proceed to configure the backend:
projects:\n - name: main\n backends:\n - type: azure\n subscription_id: 06c82ce3-28ff-4285-a146-c5e981a9d808\n tenant_id: f84a7584-88e4-4fd2-8e97-623f0a715ee1\n creds:\n type: default\nIf you don't know your subscription_id, run
az account show --query \"{subscription_id: id}\"\nThe following Azure permissions are sufficient for dstack to work:
{\n \"properties\": {\n \"roleName\": \"dstack-role\",\n \"description\": \"Minimal required permissions for using Azure with dstack\",\n \"assignableScopes\": [\n \"/subscriptions/${YOUR_SUBSCRIPTION_ID}\"\n ],\n \"permissions\": [\n {\n \"actions\": [\n \"Microsoft.Authorization/*/read\",\n \"Microsoft.Compute/availabilitySets/*\",\n \"Microsoft.Compute/locations/*\",\n \"Microsoft.Compute/virtualMachines/*\",\n \"Microsoft.Compute/virtualMachineScaleSets/*\",\n \"Microsoft.Compute/cloudServices/*\",\n \"Microsoft.Compute/disks/write\",\n \"Microsoft.Compute/disks/read\",\n \"Microsoft.Compute/disks/delete\",\n \"Microsoft.Network/networkSecurityGroups/*\",\n \"Microsoft.Network/locations/*\",\n \"Microsoft.Network/virtualNetworks/*\",\n \"Microsoft.Network/networkInterfaces/*\",\n \"Microsoft.Network/publicIPAddresses/*\",\n \"Microsoft.Resources/subscriptions/resourceGroups/read\",\n \"Microsoft.Resources/subscriptions/resourceGroups/write\",\n \"Microsoft.Resources/subscriptions/read\"\n ],\n \"notActions\": [],\n \"dataActions\": [],\n \"notDataActions\": []\n }\n ]\n }\n}\nFirst, ensure the required APIs are enabled in your GCP project_id.
PROJECT_ID=...\ngcloud config set project $PROJECT_ID\ngcloud services enable cloudapis.googleapis.com\ngcloud services enable compute.googleapis.com\nThere are two ways to configure GCP: using a service account or using the default credentials.
Service accountDefault credentialsTo create a service account, follow this guide . After setting up the service account create a key for it and download the corresponding JSON file.
Then go ahead and configure the backend by specifying the downloaded file path.
projects:\n- name: main\n backends:\n - type: gcp\n project_id: gcp-project-id\n creds:\n type: service_account\n filename: ~/.dstack/server/gcp-024ed630eab5.json\nEnable GCP application default credentials:
gcloud auth application-default login \nThen configure the backend like this:
projects:\n- name: main\n backends:\n - type: gcp\n project_id: gcp-project-id\n creds:\n type: default\nIf you don't know your GCP project ID, run
gcloud projects list --format=\"json(projectId)\"\nprojects:\n- name: main\n backends:\n - type: gcp\n project_id: gcp-project-id\n creds:\n type: default\n\n vpc_name: my-custom-vpc\nprojects:\n- name: main\n backends:\n - type: gcp\n project_id: gcp-project-id\n creds:\n type: default\n\n vpc_name: my-custom-vpc\n vpc_project_id: another-project-id\nTo use a shared VPC, that VPC has to be configured with two additional firewall rules:
INGRESS traffic on port 22, with the target tag dstack-runner-instanceINGRESS traffic on ports 22, 80, 443, with the target tag dstack-gateway-instanceThe following GCP permissions are sufficient for dstack to work:
compute.disks.create\ncompute.firewalls.create\ncompute.images.useReadOnly\ncompute.instances.create\ncompute.instances.delete\ncompute.instances.get\ncompute.instances.setLabels\ncompute.instances.setMetadata\ncompute.instances.setTags\ncompute.networks.get\ncompute.networks.updatePolicy\ncompute.regions.list\ncompute.subnetworks.list\ncompute.subnetworks.use\ncompute.subnetworks.useExternalIp\ncompute.zoneOperations.get\nIf you plan to use TPUs, additional permissions are required:
tpu.nodes.create\ntpu.nodes.delete\ntpu.nodes.get\ntpu.operations.get\ntpu.operations.list\nAlso, the use of TPUs requires the serviceAccountUser role. For TPU VMs, dstack will use the default service account.
By default, dstack utilizes public subnets and permits inbound SSH traffic exclusively for any provisioned instances. If you want dstack to use private subnets, set public_ips to false.
projects:\n - name: main\n backends:\n - type: gcp\n creds:\n type: default\n\n public_ips: false\nUsing private subnets assumes that both the dstack server and users can access the configured VPC's private subnets (e.g., through VPC peering). Additionally, Cloud NAT must be configured to provide access to external resources for provisioned instances.
There are two ways to configure OCI: using client credentials or using the default credentials.
Client credentialsDefault credentialsLog into the OCI Console , go to My profile, select API keys, and click Add API key.
Once you add a key, you'll see the configuration file. Copy its values to configure the backend as follows:
projects:\n- name: main\n backends:\n - type: oci\n creds:\n type: client\n user: ocid1.user.oc1..g5vlaeqfu47akmaafq665xsgmyaqjktyfxtacfxc4ftjxuca7aohnd2ev66m\n tenancy: ocid1.tenancy.oc1..ajqsftvk4qarcfaak3ha4ycdsaahxmaita5frdwg3tqo2bcokpd3n7oizwai\n region: eu-frankfurt-1\n fingerprint: 77:32:77:00:49:7c:cb:56:84:75:8e:77:96:7d:53:17\n key_file: ~/.oci/private_key.pem\nMake sure to include either the path to your private key via key_file or the contents of the key via key_content.
If you have default credentials set up in ~/.oci/config, configure the backend like this:
projects:\n- name: main\n backends:\n - type: oci\n creds:\n type: default\nThis is an example of a restrictive policy for a group of dstack users:
Allow group <dstack-users> to read compartments in tenancy where target.compartment.name = '<dstack-compartment>'\nAllow group <dstack-users> to read marketplace-community-listings in compartment <dstack-compartment>\nAllow group <dstack-users> to manage app-catalog-listing in compartment <dstack-compartment>\nAllow group <dstack-users> to manage instances in compartment <dstack-compartment>\nAllow group <dstack-users> to manage compute-capacity-reports in compartment <dstack-compartment>\nAllow group <dstack-users> to manage volumes in compartment <dstack-compartment>\nAllow group <dstack-users> to manage volume-attachments in compartment <dstack-compartment>\nAllow group <dstack-users> to manage virtual-network-family in compartment <dstack-compartment>\nTo use this policy, create a compartment for dstack and specify it in ~/.dstack/server/config.yml.
projects:\n- name: main\n backends:\n - type: oci\n creds:\n type: default\n compartment_id: ocid1.compartment.oc1..aaaaaaaa\nLog into your Lambda Cloud account, click API keys in the sidebar, and then click the Generate API key button to create a new API key.
Then, go ahead and configure the backend:
projects:\n- name: main\n backends:\n - type: lambda\n creds:\n type: api_key\n api_key: eersct_yrpiey-naaeedst-tk-_cb6ba38e1128464aea9bcc619e4ba2a5.iijPMi07obgt6TZ87v5qAEj61RVxhd0p\nLog into your TensorDock account, click API in the sidebar, and use the Create an Authorization section to create a new authorization key.
Then, go ahead and configure the backend:
projects:\n - name: main\n backends:\n - type: tensordock\n creds:\n type: api_key\n api_key: 248e621d-9317-7494-dc1557fa5825b-98b\n api_token: FyBI3YbnFEYXdth2xqYRnQI7hiusssBC\nThe tensordock backend supports on-demand instances only. Spot instance support coming soon.
Log into your Vast.ai account, click Account in the sidebar, and copy your API Key.
Then, go ahead and configure the backend:
projects:\n- name: main\n backends:\n - type: vastai\n creds:\n type: api_key\n api_key: d75789f22f1908e0527c78a283b523dd73051c8c7d05456516fc91e9d4efd8c5\nAlso, the vastai backend supports on-demand instances only. Spot instance support coming soon.
Log into your RunPod console, click Settings in the sidebar, expand the API Keys section, and click the button to create a key.
Then proceed to configuring the backend.
projects:\n - name: main\n backends:\n - type: runpod\n creds:\n type: api_key\n api_key: US9XTPDIV8AR42MMINY8TCKRB8S4E7LNRQ6CAUQ9\nLog into your CUDO Compute account, click API keys in the sidebar, and click the Create an API key button.
Ensure you've created a project with CUDO Compute, then proceed to configuring the backend.
projects:\n - name: main\n backends:\n - type: cudo\n project_id: my-cudo-project\n creds:\n type: api_key\n api_key: 7487240a466624b48de22865589\nLog into your DataCrunch account, click Account Settings in the sidebar, find REST API Credentials area and then click the Generate Credentials button.
Then, go ahead and configure the backend:
projects:\n - name: main\n backends:\n - type: datacrunch\n creds:\n type: api_key\n client_id: xfaHBqYEsArqhKWX-e52x3HH7w8T\n client_secret: B5ZU5Qx9Nt8oGMlmMhNI3iglK8bjMhagTbylZy4WzncZe39995f7Vxh8\ndstack supports both self-managed, and managed Kubernetes clusters.
To use GPUs with Kubernetes, the cluster must be installed with the NVIDIA GPU Operator .
To configure a Kubernetes backend, specify the path to the kubeconfig file, and the port that dstack can use for proxying SSH traffic. In case of a self-managed cluster, also specify the IP address of any node in the cluster.
Here's how to configure the backend to use a self-managed cluster.
projects:\n- name: main\n backends:\n - type: kubernetes\n kubeconfig:\n filename: ~/.kube/config\n networking:\n ssh_host: localhost # The external IP address of any node\n ssh_port: 32000 # Any port accessible outside of the cluster\nThe port specified to ssh_port must be accessible outside of the cluster.
If you are using Kind, make sure to make to set up ssh_port via extraPortMappings for proxying SSH traffic:
kind: Cluster\napiVersion: kind.x-k8s.io/v1alpha4\nnodes:\n - role: control-plane\n extraPortMappings:\n - containerPort: 32000 # Must be same as `ssh_port`\n hostPort: 32000 # Must be same as `ssh_port`\nGo ahead and create the cluster like this:
kind create cluster --config examples/misc/kubernetes/kind-config.yml\nHere's how to configure the backend to use a managed cluster (AWS, GCP, Azure).
projects:\n - name: main\n backends:\n - type: kubernetes\n kubeconfig:\n filename: ~/.kube/config\n networking:\n ssh_port: 32000 # Any port accessible outside of the cluster\nThe port specified to ssh_port must be accessible outside of the cluster.
For example, if you are using EKS, make sure to add it via an ingress rule of the corresponding security group:
aws ec2 authorize-security-group-ingress --group-id <cluster-security-group-id> --protocol tcp --port 32000 --cidr 0.0.0.0/0\nprojects - The list of projects.","text":""},{"location":"docs/reference/server/config.yml/#projects","title":"projects[n]","text":""},{"location":"docs/reference/server/config.yml/#name","title":"name - The name of the project.","text":""},{"location":"docs/reference/server/config.yml/#backends","title":"backends - The list of backends.","text":""},{"location":"docs/reference/server/config.yml/#aws","title":"projects[n].backends[type=aws]","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of the backend. Must be aws.","text":""},{"location":"docs/reference/server/config.yml/#regions","title":"regions - (Optional) The list of AWS regions.","text":""},{"location":"docs/reference/server/config.yml/#vpc_name","title":"vpc_name - (Optional) The VPC name. All configured regions must have a VPC with this name.","text":""},{"location":"docs/reference/server/config.yml/#vpc_ids","title":"vpc_ids - (Optional) The mapping from AWS regions to VPC IDs. If default_vpcs: true, omitted regions will use default VPCs.","text":""},{"location":"docs/reference/server/config.yml/#default_vpcs","title":"default_vpcs - (Optional) A flag to enable/disable using default VPCs in regions not configured by vpc_ids. Set to false if default VPCs should never be used. Defaults to true.","text":""},{"location":"docs/reference/server/config.yml/#public_ips","title":"public_ips - (Optional) A flag to enable/disable public IP assigning on instances. Defaults to true.","text":""},{"location":"docs/reference/server/config.yml/#_creds","title":"creds - The credentials.","text":""},{"location":"docs/reference/server/config.yml/#aws-creds","title":"projects[n].backends[type=aws].creds","text":"Access keyDefault"},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be access_key.","text":""},{"location":"docs/reference/server/config.yml/#access_key","title":"access_key - The access key.","text":""},{"location":"docs/reference/server/config.yml/#secret_key","title":"secret_key - The secret key.","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be default.","text":""},{"location":"docs/reference/server/config.yml/#azure","title":"projects[n].backends[type=azure]","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of the backend. Must be azure.","text":""},{"location":"docs/reference/server/config.yml/#tenant_id","title":"tenant_id - The tenant ID.","text":""},{"location":"docs/reference/server/config.yml/#subscription_id","title":"subscription_id - The subscription ID.","text":""},{"location":"docs/reference/server/config.yml/#_creds","title":"creds - The credentials.","text":""},{"location":"docs/reference/server/config.yml/#azure-creds","title":"projects[n].backends[type=azure].creds","text":"ClientDefault"},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be client.","text":""},{"location":"docs/reference/server/config.yml/#client_id","title":"client_id - The client ID.","text":""},{"location":"docs/reference/server/config.yml/#client_secret","title":"client_secret - The client secret.","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be default.","text":""},{"location":"docs/reference/server/config.yml/#datacrunch","title":"projects[n].backends[type=datacrunch]","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of backend. Must be datacrunch.","text":""},{"location":"docs/reference/server/config.yml/#_creds","title":"creds - The credentials.","text":""},{"location":"docs/reference/server/config.yml/#datacrunch-creds","title":"projects[n].backends[type=datacrunch].creds","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be api_key.","text":""},{"location":"docs/reference/server/config.yml/#client_id","title":"client_id - The client ID.","text":""},{"location":"docs/reference/server/config.yml/#client_secret","title":"client_secret - The client secret.","text":""},{"location":"docs/reference/server/config.yml/#gcp","title":"projects[n].backends[type=gcp]","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of backend. Must be gcp.","text":""},{"location":"docs/reference/server/config.yml/#project_id","title":"project_id - The project ID.","text":""},{"location":"docs/reference/server/config.yml/#vpc_name","title":"vpc_name - (Optional) The VPC name.","text":""},{"location":"docs/reference/server/config.yml/#vpc_project_id","title":"vpc_project_id - (Optional) The shared VPC hosted project ID. Required for shared VPC only.","text":""},{"location":"docs/reference/server/config.yml/#public_ips","title":"public_ips - (Optional) A flag to enable/disable public IP assigning on instances. Defaults to true.","text":""},{"location":"docs/reference/server/config.yml/#_creds","title":"creds - The credentials.","text":""},{"location":"docs/reference/server/config.yml/#gcp-creds","title":"projects[n].backends[type=gcp].creds","text":"Service accountDefault"},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be service_account.","text":""},{"location":"docs/reference/server/config.yml/#filename","title":"filename - The path to the service account file.","text":""},{"location":"docs/reference/server/config.yml/#data","title":"data - (Optional) The contents of the service account file.","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be default.","text":""},{"location":"docs/reference/server/config.yml/#lambda","title":"projects[n].backends[type=lambda]","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of backend. Must be lambda.","text":""},{"location":"docs/reference/server/config.yml/#_creds","title":"creds - The credentials.","text":""},{"location":"docs/reference/server/config.yml/#lambda-creds","title":"projects[n].backends[type=lambda].creds","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be api_key.","text":""},{"location":"docs/reference/server/config.yml/#api_key","title":"api_key - The API key.","text":""},{"location":"docs/reference/server/config.yml/#oci","title":"projects[n].backends[type=oci]","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of backend. Must be oci.","text":""},{"location":"docs/reference/server/config.yml/#_creds","title":"creds - The credentials.","text":""},{"location":"docs/reference/server/config.yml/#regions","title":"regions - (Optional) List of region names for running dstack jobs. Omit to use all regions.","text":""},{"location":"docs/reference/server/config.yml/#compartment_id","title":"compartment_id - (Optional) Compartment where dstack will create all resources. Omit to instruct dstack to create a new compartment.","text":""},{"location":"docs/reference/server/config.yml/#oci-creds","title":"projects[n].backends[type=oci].creds","text":"ClientDefault"},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be client.","text":""},{"location":"docs/reference/server/config.yml/#user","title":"user - User OCID.","text":""},{"location":"docs/reference/server/config.yml/#tenancy","title":"tenancy - Tenancy OCID.","text":""},{"location":"docs/reference/server/config.yml/#key_file","title":"key_file - (Optional) Path to the user's private PEM key. Either this or key_content should be set.","text":""},{"location":"docs/reference/server/config.yml/#key_content","title":"key_content - (Optional) Content of the user's private PEM key. Either this or key_file should be set.","text":""},{"location":"docs/reference/server/config.yml/#pass_phrase","title":"pass_phrase - (Optional) Passphrase for the private PEM key if it is encrypted.","text":""},{"location":"docs/reference/server/config.yml/#fingerprint","title":"fingerprint - User's public key fingerprint.","text":""},{"location":"docs/reference/server/config.yml/#region","title":"region - Name or key of any region the tenancy is subscribed to.","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be default.","text":""},{"location":"docs/reference/server/config.yml/#file","title":"file - (Optional) Path to the OCI CLI-compatible config file. Defaults to ~/.oci/config.","text":""},{"location":"docs/reference/server/config.yml/#profile","title":"profile - (Optional) Profile to load from the config file. Defaults to DEFAULT.","text":""},{"location":"docs/reference/server/config.yml/#tensordock","title":"projects[n].backends[type=tensordock]","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of backend. Must be tensordock.","text":""},{"location":"docs/reference/server/config.yml/#_creds","title":"creds - The credentials.","text":""},{"location":"docs/reference/server/config.yml/#tensordock-creds","title":"projects[n].backends[type=tensordock].creds","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be api_key.","text":""},{"location":"docs/reference/server/config.yml/#api_key","title":"api_key - The API key.","text":""},{"location":"docs/reference/server/config.yml/#api_token","title":"api_token - The API token.","text":""},{"location":"docs/reference/server/config.yml/#vastai","title":"projects[n].backends[type=vastai]","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of backend. Must be vastai.","text":""},{"location":"docs/reference/server/config.yml/#_creds","title":"creds - The credentials.","text":""},{"location":"docs/reference/server/config.yml/#vastai-creds","title":"projects[n].backends[type=vastai].creds","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be api_key.","text":""},{"location":"docs/reference/server/config.yml/#api_key","title":"api_key - The API key.","text":""},{"location":"docs/reference/server/config.yml/#cudo","title":"projects[n].backends[type=cudo]","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of backend. Must be cudo.","text":""},{"location":"docs/reference/server/config.yml/#project_id","title":"project_id - The project ID.","text":""},{"location":"docs/reference/server/config.yml/#_creds","title":"creds - The credentials.","text":""},{"location":"docs/reference/server/config.yml/#cudo-creds","title":"projects[n].backends[type=cudo].creds","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of credentials. Must be api_key.","text":""},{"location":"docs/reference/server/config.yml/#api_key","title":"api_key - The API key.","text":""},{"location":"docs/reference/server/config.yml/#kubernetes","title":"projects[n].backends[type=kubernetes]","text":""},{"location":"docs/reference/server/config.yml/#type","title":"type - The type of backend. Must be kubernetes.","text":""},{"location":"docs/reference/server/config.yml/#_kubeconfig","title":"kubeconfig - The kubeconfig configuration.","text":""},{"location":"docs/reference/server/config.yml/#_networking","title":"networking - (Optional) The networking configuration.","text":""},{"location":"docs/reference/server/config.yml/#kubeconfig","title":"projects[n].backends[type=kubernetes].kubeconfig","text":""},{"location":"docs/reference/server/config.yml/#filename","title":"filename - The path to the kubeconfig file.","text":""},{"location":"docs/reference/server/config.yml/#data","title":"data - (Optional) The contents of the kubeconfig file.","text":""},{"location":"docs/reference/server/config.yml/#networking","title":"projects[n].backends[type=kubernetes].networking","text":""},{"location":"docs/reference/server/config.yml/#ssh_host","title":"ssh_host - (Optional) The external IP address of any node.","text":""},{"location":"docs/reference/server/config.yml/#ssh_port","title":"ssh_port - (Optional) Any port accessible outside of the cluster.","text":""},{"location":"blog/archive/2024/","title":"2024","text":""}]}
\ No newline at end of file