diff --git a/index.html b/index.html index 121f862..c76b5be 100644 --- a/index.html +++ b/index.html @@ -187,5 +187,5 @@

Getting Help diff --git a/search/search_index.json b/search/search_index.json index c0a8589..65acbc8 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Welcome to FarmShare \u00b6 FarmShare is Stanford\u2019s community computing environment. It is intended for use in coursework and unsponsored research. It is not approved for use with high-risk data, or for use in sponsored research. FarmShare evolved from the old, public UNIX cluster, once located on the second floor of Sweet Hall, which was itself a successor to systems like the University\u2019s original timeshare service, LOTS. The latest iteration of the FarmShare environment came online early 2024. Resources on Farmshare are focused on making it easier to learn how to use research computing including \u201cscheduler\u201d or \u201cdistributed resource management system\u201d to submit compute jobs. By using FarmShare, new researchers can more easily adapt to using larger clusters when they have big projects that involve using federally funded resources, shared Stanford clusters, or on a small grant funded cluster. Full SUNet (or sponsorship) required. Cluster Components \u00b6 FarmShare consists of three classes of servers: The rice servers are login nodes where you log in to run commands, access files, submit jobs, and review results. The rice servers also have access to Stanford AFS . These servers can be used for interactive work, but some resource limits are enforced, so if you need to run a long-running or compute- and/or memory-intensive process you should submit a job. Remember, these are servers for research and academic use, not for administrative or business functions. The iron , rye and wheat servers are compute nodes. They have more CPU power and more memory than the rice servers, and are meant for both interactive jobs (where you log in to control what happens) and batch jobs (where everything runs from a script that you submit). Some wheat and iron nodes also have significantly more memory than others. Like the rice servers, these are available for use for coursework or research purposes. The oat servers are GPU compute nodes. They are similar to the compute nodes mentioned above, except that they also have GPUs installed. These nodes are meant for computational academic or research work that is able to take advantage of a GPU (e.g., TensorFlow jobs fit into this category). All FarmShare servers run Ubuntu 22.04 LTS FarmShare currently has 4 rice servers (login nodes) along with 26 compute nodes (wheat, iron, oat nodes). Eligibility \u00b6 FarmShare is available to anyone who has a full-service SUNetID. A full-service SUNetID is one that has email service; if you can successfully get to Stanford Webmail , then you are eligible to use FarmShare 2 for academic work and small research jobs! If you do not already have a full-service SUNetID (maybe because you are a visiting researcher), you can get a sponsored full-service SUNetID. Read more about SUNetID levels . Note that, in order to get a sponsored SUNetID, a monthly fee will be charged by University IT. Only people with spending authority may sponsor a SUNetID. Sponsorships can be obtained and paid for through Sponsorship Manager . Current rates are available from the Sponsored Account Rates page . FarmShare is meant for low- or moderate-risk data , and is primarily intended for class work and for other general- and personal-use work (research and training). It is not meant for sponsored research (where you have a dedicated source of funding), and is not approved for handling high-risk data. Sponsored Research \u00b6 If you are doing sponsored or departmental research, then FarmShare might not be the right place for you. Instead, if the data you are working with is all low-risk, then you should consider getting access to our Sherlock Cluster! The Sherlock web site has more information about how to get access. Getting Help \u00b6 Most FarmShare support is provided during business hours, either via email or during academic-year office hours. For email support, send a message to srcc-support . Make sure you have \u201cFarmShare\u201d somewhere in the subject line, and please be as detailed as possible with your request.","title":"Introduction"},{"location":"#welcome-to-farmshare","text":"FarmShare is Stanford\u2019s community computing environment. It is intended for use in coursework and unsponsored research. It is not approved for use with high-risk data, or for use in sponsored research. FarmShare evolved from the old, public UNIX cluster, once located on the second floor of Sweet Hall, which was itself a successor to systems like the University\u2019s original timeshare service, LOTS. The latest iteration of the FarmShare environment came online early 2024. Resources on Farmshare are focused on making it easier to learn how to use research computing including \u201cscheduler\u201d or \u201cdistributed resource management system\u201d to submit compute jobs. By using FarmShare, new researchers can more easily adapt to using larger clusters when they have big projects that involve using federally funded resources, shared Stanford clusters, or on a small grant funded cluster. Full SUNet (or sponsorship) required.","title":"Welcome to FarmShare"},{"location":"#cluster-components","text":"FarmShare consists of three classes of servers: The rice servers are login nodes where you log in to run commands, access files, submit jobs, and review results. The rice servers also have access to Stanford AFS . These servers can be used for interactive work, but some resource limits are enforced, so if you need to run a long-running or compute- and/or memory-intensive process you should submit a job. Remember, these are servers for research and academic use, not for administrative or business functions. The iron , rye and wheat servers are compute nodes. They have more CPU power and more memory than the rice servers, and are meant for both interactive jobs (where you log in to control what happens) and batch jobs (where everything runs from a script that you submit). Some wheat and iron nodes also have significantly more memory than others. Like the rice servers, these are available for use for coursework or research purposes. The oat servers are GPU compute nodes. They are similar to the compute nodes mentioned above, except that they also have GPUs installed. These nodes are meant for computational academic or research work that is able to take advantage of a GPU (e.g., TensorFlow jobs fit into this category). All FarmShare servers run Ubuntu 22.04 LTS FarmShare currently has 4 rice servers (login nodes) along with 26 compute nodes (wheat, iron, oat nodes).","title":"Cluster Components"},{"location":"#eligibility","text":"FarmShare is available to anyone who has a full-service SUNetID. A full-service SUNetID is one that has email service; if you can successfully get to Stanford Webmail , then you are eligible to use FarmShare 2 for academic work and small research jobs! If you do not already have a full-service SUNetID (maybe because you are a visiting researcher), you can get a sponsored full-service SUNetID. Read more about SUNetID levels . Note that, in order to get a sponsored SUNetID, a monthly fee will be charged by University IT. Only people with spending authority may sponsor a SUNetID. Sponsorships can be obtained and paid for through Sponsorship Manager . Current rates are available from the Sponsored Account Rates page . FarmShare is meant for low- or moderate-risk data , and is primarily intended for class work and for other general- and personal-use work (research and training). It is not meant for sponsored research (where you have a dedicated source of funding), and is not approved for handling high-risk data.","title":"Eligibility"},{"location":"#sponsored-research","text":"If you are doing sponsored or departmental research, then FarmShare might not be the right place for you. Instead, if the data you are working with is all low-risk, then you should consider getting access to our Sherlock Cluster! The Sherlock web site has more information about how to get access.","title":"Sponsored Research"},{"location":"#getting-help","text":"Most FarmShare support is provided during business hours, either via email or during academic-year office hours. For email support, send a message to srcc-support . Make sure you have \u201cFarmShare\u201d somewhere in the subject line, and please be as detailed as possible with your request.","title":"Getting Help"},{"location":"capacity/","text":"Adding Capacity \u00b6 FarmShare does not have a dedicated funding source available to it, and we appreciate any contributions that people can make. For example: If you are using FarmShare for a class, let your department chair know that you are using it. If you are a registered student group using FarmShare, let your faculty representative and/or ASSU know that you are using it. If you have spare funds available, email us and we can arrange either a iJournals transfer to our PTA, or a capital purchase from your PTA. If you already have hardware being supported by SRCC, we may be able to use that hardware once you are done with it. We appreciate anything you can do to get the word out about FarmShare and how awesome it is!","title":"Adding Capacity"},{"location":"capacity/#adding-capacity","text":"FarmShare does not have a dedicated funding source available to it, and we appreciate any contributions that people can make. For example: If you are using FarmShare for a class, let your department chair know that you are using it. If you are a registered student group using FarmShare, let your faculty representative and/or ASSU know that you are using it. If you have spare funds available, email us and we can arrange either a iJournals transfer to our PTA, or a capital purchase from your PTA. If you already have hardware being supported by SRCC, we may be able to use that hardware once you are done with it. We appreciate anything you can do to get the word out about FarmShare and how awesome it is!","title":"Adding Capacity"},{"location":"connecting/","text":"Getting Connected \u00b6 SSH on Linux and macOS \u00b6 SSH provides a secure, remote terminal connection to the login nodes. There are a number of rice servers providing login service (e.g., rice-01, rice-02); the easiest way to connect is using the load-balanced name, login.farmshare.stanford.edu , which will select one for you according to recent utilization. Simply open up your favorite terminal program (on macOS you can use the built-in Terminal application or something like iTerm) and run the ssh command like so, replacing SUNetID with your own SUNet ID: ssh SUNetID@login.farmshare.stanford.edu Depending on your environment and configuration you may be prompted for your SUNet ID password, and you\u2019ll need to complete your login using two-step authentication . SSH on Windows \u00b6 Windows does not have a built-in SSH client, but a number are available from third parties: MobaXterm \u00b6 MobaXterm is an SSH client with a built-in X server, making remote display extremely convenient, and the Home Edition is free for personal use. PuTTY \u00b6 PuTTY is a popular, freely available SSH client for Windows. The default settings are appropriate for most users, so all you need to do is specify the host name and click the Open button to connect. Aat the login as: prompt, enter your SUNet ID, and authenticate by password if asked. A number of configuration options are available, and PuTTY allows you to customize your connection settings and save them as a session. The next time you start the application you can load the saved configuration by selecting the desired session and clicking the Load button. PuTTY supports GSSAPI authentication as of version 0.61 and will automatically use your existing Kerberos credentials for login, if available. You may also want to enable Kerberos ticket forwarding by checking the Connection \u2192 SSH \u2192 Auth \u2192 GSSAPI \u2192 Allow GSSAPI credential delegation box. Mobile Shell (Mosh) \u00b6 Mosh is an alternative to SSH for Linux and macOS clients. It uses SSH for authentication, so you may want to review the suggested SSH configuration above. Mosh has some advantages, including predictive display, which can be useful on high-latency connections, and improved network resiliency. Mosh connections can persist when you switch networks and can even survive putting your computer to sleep. OnDemand \u00b6 Open OnDemand is a browser based interface to FarmShare. OnDemand offers terminal, file manager, editor, desktop, and GUI applications right from your web browser!","title":"Getting Connected"},{"location":"connecting/#getting-connected","text":"","title":"Getting Connected"},{"location":"connecting/#ssh-on-linux-and-macos","text":"SSH provides a secure, remote terminal connection to the login nodes. There are a number of rice servers providing login service (e.g., rice-01, rice-02); the easiest way to connect is using the load-balanced name, login.farmshare.stanford.edu , which will select one for you according to recent utilization. Simply open up your favorite terminal program (on macOS you can use the built-in Terminal application or something like iTerm) and run the ssh command like so, replacing SUNetID with your own SUNet ID: ssh SUNetID@login.farmshare.stanford.edu Depending on your environment and configuration you may be prompted for your SUNet ID password, and you\u2019ll need to complete your login using two-step authentication .","title":"SSH on Linux and macOS"},{"location":"connecting/#ssh-on-windows","text":"Windows does not have a built-in SSH client, but a number are available from third parties:","title":"SSH on Windows"},{"location":"connecting/#mobaxterm","text":"MobaXterm is an SSH client with a built-in X server, making remote display extremely convenient, and the Home Edition is free for personal use.","title":"MobaXterm"},{"location":"connecting/#putty","text":"PuTTY is a popular, freely available SSH client for Windows. The default settings are appropriate for most users, so all you need to do is specify the host name and click the Open button to connect. Aat the login as: prompt, enter your SUNet ID, and authenticate by password if asked. A number of configuration options are available, and PuTTY allows you to customize your connection settings and save them as a session. The next time you start the application you can load the saved configuration by selecting the desired session and clicking the Load button. PuTTY supports GSSAPI authentication as of version 0.61 and will automatically use your existing Kerberos credentials for login, if available. You may also want to enable Kerberos ticket forwarding by checking the Connection \u2192 SSH \u2192 Auth \u2192 GSSAPI \u2192 Allow GSSAPI credential delegation box.","title":"PuTTY"},{"location":"connecting/#mobile-shell-mosh","text":"Mosh is an alternative to SSH for Linux and macOS clients. It uses SSH for authentication, so you may want to review the suggested SSH configuration above. Mosh has some advantages, including predictive display, which can be useful on high-latency connections, and improved network resiliency. Mosh connections can persist when you switch networks and can even survive putting your computer to sleep.","title":"Mobile Shell (Mosh)"},{"location":"connecting/#ondemand","text":"Open OnDemand is a browser based interface to FarmShare. OnDemand offers terminal, file manager, editor, desktop, and GUI applications right from your web browser!","title":"OnDemand"},{"location":"slurm/","text":"Running Jobs \u00b6 Slurm \u00b6 FarmShare uses Slurm for job (resource) management. Full documentation and detailed usage information is provided in the man pages for the srun, sbatch, squeue, scancel, sinfo, and scontrol commands. Jobs are scheduled according to a priority which depends on a number of factors, including how long a job has been waiting, its size, and a fair-share value that tracks recent per-user utilization of cluster resources. Lower-priority jobs, and jobs requiring access to resources not currently available, may wait some time before starting to run. The scheduler may reserve resources so that pending jobs can start; while it will try to backfill these resources with smaller, shorter jobs (even those at lower priorities), this behavior can sometimes cause nodes to appear to be idle even when there are jobs that are ready to run. You can use squeue --start to get an estimate of when pending jobs will start. Interactive Jobs \u00b6 Interactive sessions that require resources in excess of limits on the login nodes, exclusive access to resources, or access to a feature not available on the login nodes (e.g., a GPU), can be submitted to a compute node. Each user is allowed one interactive job, which may run for at most one day. You can use the srun command to request one: ta5@rice-04:~$ srun --qos=interactive --pty bash ta5@wheat-01:~$ Batch Jobs \u00b6 The sbatch command is used to submit a batch job, and takes a batch script as an argument. Options are used to request specific resources (including runtime), and can be provided either on the command line or, using a special syntax, in the script file itself. sbatch can also be used to submit many similar jobs, each perhaps varying in only one or two parameters, in a single invocation using the \u2013array option; each job in an array has access to environment variables identifying its rank. Sample batch script to submit a job: ta5@rice-04:~$ cat hello_world.sh #!/bin/bash #SBATCH --job-name=hello_world #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=1 #SBATCH --partition=normal echo 'Hello World!' ta5@rice-04:~$ ta5@rice-04:~$ ta5@rice-04:~$ sbatch hello_world.sh Submitted batch job 177987 ta5@rice-04:~$ ta5@rice-04:~$ cat slurm-177987.out Hello World! ta5@rice-04:~$ Partition Info \u00b6 Partition Max Memory Max CPUs Time Limit normal 188GB 256 2 Days","title":"Running Jobs"},{"location":"slurm/#running-jobs","text":"","title":"Running Jobs"},{"location":"slurm/#slurm","text":"FarmShare uses Slurm for job (resource) management. Full documentation and detailed usage information is provided in the man pages for the srun, sbatch, squeue, scancel, sinfo, and scontrol commands. Jobs are scheduled according to a priority which depends on a number of factors, including how long a job has been waiting, its size, and a fair-share value that tracks recent per-user utilization of cluster resources. Lower-priority jobs, and jobs requiring access to resources not currently available, may wait some time before starting to run. The scheduler may reserve resources so that pending jobs can start; while it will try to backfill these resources with smaller, shorter jobs (even those at lower priorities), this behavior can sometimes cause nodes to appear to be idle even when there are jobs that are ready to run. You can use squeue --start to get an estimate of when pending jobs will start.","title":"Slurm"},{"location":"slurm/#interactive-jobs","text":"Interactive sessions that require resources in excess of limits on the login nodes, exclusive access to resources, or access to a feature not available on the login nodes (e.g., a GPU), can be submitted to a compute node. Each user is allowed one interactive job, which may run for at most one day. You can use the srun command to request one: ta5@rice-04:~$ srun --qos=interactive --pty bash ta5@wheat-01:~$","title":"Interactive Jobs"},{"location":"slurm/#batch-jobs","text":"The sbatch command is used to submit a batch job, and takes a batch script as an argument. Options are used to request specific resources (including runtime), and can be provided either on the command line or, using a special syntax, in the script file itself. sbatch can also be used to submit many similar jobs, each perhaps varying in only one or two parameters, in a single invocation using the \u2013array option; each job in an array has access to environment variables identifying its rank. Sample batch script to submit a job: ta5@rice-04:~$ cat hello_world.sh #!/bin/bash #SBATCH --job-name=hello_world #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=1 #SBATCH --partition=normal echo 'Hello World!' ta5@rice-04:~$ ta5@rice-04:~$ ta5@rice-04:~$ sbatch hello_world.sh Submitted batch job 177987 ta5@rice-04:~$ ta5@rice-04:~$ cat slurm-177987.out Hello World! ta5@rice-04:~$","title":"Batch Jobs"},{"location":"slurm/#partition-info","text":"Partition Max Memory Max CPUs Time Limit normal 188GB 256 2 Days","title":"Partition Info"},{"location":"software/","text":"Using Software \u00b6 FarmShare has lots of free and commercial software available to use. We provide software that comes with Ubuntu, software that we package ourselves, and we also provide the capability for you to build and use software yourself. Software Sources \u00b6 Software on FarmShare comes from three sources: Packages Modules Build your own Packages \u00b6 Packaged software is easiest to use, because you don\u2019t have to do anything. Packaged software has already been installed on all of the systems in the environment, so to use the software, you just have to run the command. For example, to run the packaged version of Python, you just need to run the python3 command: ta5@rice-01:~$ python3 --version Python 3.10.12 FarmShare 2 runs Ubuntu 22.04 LTS, which means almost anything in the Ubuntu Jammy package repository is available to be installed. Modules \u00b6 To provide up-to-date software, we use modules. Modularized software lives in shared, network-connected storage, and is built by the FarmShare support team. To see a listing of available software, run the module available command: ta5@rice-01:~$ module available ---------------------------- /software/modules/linux-ubuntu22.04-x86_64/Core ---------------------------- apptainer/1.1.9 libpng/1.5.30 blast-plus/2.14.1 llvm/16.0.6 boost/1.83.0 micromamba/1.1.0 bowtie2/2.5.1 mpich/4.1.2 cuda/11.4.4 numactl/2.0.14 cudnn/8.2.4.15-11.4 openblas/0.3.23 gcc/13.2.0 openblas/0.3.24 (D) imagemagick/7.1.1-11 openmpi/1.10.7 intel-oneapi-compilers/2023.2.1 py-jupyterlab/4.0.1_python/3.11.6 intel-oneapi-dal/2023.2.0 py-pip/23.1.2_python/3.11.6 intel-oneapi-ipp/2021.9.0 python/3.11.6 intel-oneapi-mkl/2023.2.0 r-magick/2.7.4_r/4.3.0 intel-oneapi-mpi/2021.10.0 r-tidyverse/2.0.0_r/4.3.0 intel-oneapi-tbb/2021.10.0 r/4.3.0 julia/1.9.2 rust/1.75.0 libjpeg-turbo/2.1.5.1 ------------------------------------- /software/modules/commercial -------------------------------------- mathematica/13.3.1 mathematica/14.0.0 (D) matlab/r2023b sas/9.4 stata/18 Where: D: Default Module If the avail list is too long consider trying: \"module --default avail\" or \"ml -d av\" to just list the default modules. \"module overview\" or \"ml ov\" to display the number of modules for each name. Use \"module spider\" to find all possible modules and extensions. Use \"module keyword key1 key2 ...\" to search for all possible modules matching any of the \"keys\". Build Your Own \u00b6 In addition to the software that we provide for you to use, it is perfectly OK for you to build and use your own software.","title":"Using Software"},{"location":"software/#using-software","text":"FarmShare has lots of free and commercial software available to use. We provide software that comes with Ubuntu, software that we package ourselves, and we also provide the capability for you to build and use software yourself.","title":"Using Software"},{"location":"software/#software-sources","text":"Software on FarmShare comes from three sources: Packages Modules Build your own","title":"Software Sources"},{"location":"software/#packages","text":"Packaged software is easiest to use, because you don\u2019t have to do anything. Packaged software has already been installed on all of the systems in the environment, so to use the software, you just have to run the command. For example, to run the packaged version of Python, you just need to run the python3 command: ta5@rice-01:~$ python3 --version Python 3.10.12 FarmShare 2 runs Ubuntu 22.04 LTS, which means almost anything in the Ubuntu Jammy package repository is available to be installed.","title":"Packages"},{"location":"software/#modules","text":"To provide up-to-date software, we use modules. Modularized software lives in shared, network-connected storage, and is built by the FarmShare support team. To see a listing of available software, run the module available command: ta5@rice-01:~$ module available ---------------------------- /software/modules/linux-ubuntu22.04-x86_64/Core ---------------------------- apptainer/1.1.9 libpng/1.5.30 blast-plus/2.14.1 llvm/16.0.6 boost/1.83.0 micromamba/1.1.0 bowtie2/2.5.1 mpich/4.1.2 cuda/11.4.4 numactl/2.0.14 cudnn/8.2.4.15-11.4 openblas/0.3.23 gcc/13.2.0 openblas/0.3.24 (D) imagemagick/7.1.1-11 openmpi/1.10.7 intel-oneapi-compilers/2023.2.1 py-jupyterlab/4.0.1_python/3.11.6 intel-oneapi-dal/2023.2.0 py-pip/23.1.2_python/3.11.6 intel-oneapi-ipp/2021.9.0 python/3.11.6 intel-oneapi-mkl/2023.2.0 r-magick/2.7.4_r/4.3.0 intel-oneapi-mpi/2021.10.0 r-tidyverse/2.0.0_r/4.3.0 intel-oneapi-tbb/2021.10.0 r/4.3.0 julia/1.9.2 rust/1.75.0 libjpeg-turbo/2.1.5.1 ------------------------------------- /software/modules/commercial -------------------------------------- mathematica/13.3.1 mathematica/14.0.0 (D) matlab/r2023b sas/9.4 stata/18 Where: D: Default Module If the avail list is too long consider trying: \"module --default avail\" or \"ml -d av\" to just list the default modules. \"module overview\" or \"ml ov\" to display the number of modules for each name. Use \"module spider\" to find all possible modules and extensions. Use \"module keyword key1 key2 ...\" to search for all possible modules matching any of the \"keys\".","title":"Modules"},{"location":"software/#build-your-own","text":"In addition to the software that we provide for you to use, it is perfectly OK for you to build and use your own software.","title":"Build Your Own"}]} \ No newline at end of file +{"config":{"lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Welcome to FarmShare \u00b6 FarmShare is Stanford\u2019s community computing environment. It is intended for use in coursework and unsponsored research. It is not approved for use with high-risk data, or for use in sponsored research. FarmShare evolved from the old, public UNIX cluster, once located on the second floor of Sweet Hall, which was itself a successor to systems like the University\u2019s original timeshare service, LOTS. The latest iteration of the FarmShare environment came online early 2024. Resources on Farmshare are focused on making it easier to learn how to use research computing including \u201cscheduler\u201d or \u201cdistributed resource management system\u201d to submit compute jobs. By using FarmShare, new researchers can more easily adapt to using larger clusters when they have big projects that involve using federally funded resources, shared Stanford clusters, or on a small grant funded cluster. Full SUNet (or sponsorship) required. Cluster Components \u00b6 FarmShare consists of three classes of servers: The rice servers are login nodes where you log in to run commands, access files, submit jobs, and review results. The rice servers also have access to Stanford AFS . These servers can be used for interactive work, but some resource limits are enforced, so if you need to run a long-running or compute- and/or memory-intensive process you should submit a job. Remember, these are servers for research and academic use, not for administrative or business functions. The iron , rye and wheat servers are compute nodes. They have more CPU power and more memory than the rice servers, and are meant for both interactive jobs (where you log in to control what happens) and batch jobs (where everything runs from a script that you submit). Some wheat and iron nodes also have significantly more memory than others. Like the rice servers, these are available for use for coursework or research purposes. The oat servers are GPU compute nodes. They are similar to the compute nodes mentioned above, except that they also have GPUs installed. These nodes are meant for computational academic or research work that is able to take advantage of a GPU (e.g., TensorFlow jobs fit into this category). All FarmShare servers run Ubuntu 22.04 LTS FarmShare currently has 4 rice servers (login nodes) along with 26 compute nodes (wheat, iron, oat nodes). Eligibility \u00b6 FarmShare is available to anyone who has a full-service SUNetID. A full-service SUNetID is one that has email service; if you can successfully get to Stanford Webmail , then you are eligible to use FarmShare 2 for academic work and small research jobs! If you do not already have a full-service SUNetID (maybe because you are a visiting researcher), you can get a sponsored full-service SUNetID. Read more about SUNetID levels . Note that, in order to get a sponsored SUNetID, a monthly fee will be charged by University IT. Only people with spending authority may sponsor a SUNetID. Sponsorships can be obtained and paid for through Sponsorship Manager . Current rates are available from the Sponsored Account Rates page . FarmShare is meant for low- or moderate-risk data , and is primarily intended for class work and for other general- and personal-use work (research and training). It is not meant for sponsored research (where you have a dedicated source of funding), and is not approved for handling high-risk data. Sponsored Research \u00b6 If you are doing sponsored or departmental research, then FarmShare might not be the right place for you. Instead, if the data you are working with is all low-risk, then you should consider getting access to our Sherlock Cluster! The Sherlock web site has more information about how to get access. Getting Help \u00b6 Most FarmShare support is provided during business hours, either via email or during academic-year office hours. For email support, send a message to srcc-support . Make sure you have \u201cFarmShare\u201d somewhere in the subject line, and please be as detailed as possible with your request.","title":"Introduction"},{"location":"#welcome-to-farmshare","text":"FarmShare is Stanford\u2019s community computing environment. It is intended for use in coursework and unsponsored research. It is not approved for use with high-risk data, or for use in sponsored research. FarmShare evolved from the old, public UNIX cluster, once located on the second floor of Sweet Hall, which was itself a successor to systems like the University\u2019s original timeshare service, LOTS. The latest iteration of the FarmShare environment came online early 2024. Resources on Farmshare are focused on making it easier to learn how to use research computing including \u201cscheduler\u201d or \u201cdistributed resource management system\u201d to submit compute jobs. By using FarmShare, new researchers can more easily adapt to using larger clusters when they have big projects that involve using federally funded resources, shared Stanford clusters, or on a small grant funded cluster. Full SUNet (or sponsorship) required.","title":"Welcome to FarmShare"},{"location":"#cluster-components","text":"FarmShare consists of three classes of servers: The rice servers are login nodes where you log in to run commands, access files, submit jobs, and review results. The rice servers also have access to Stanford AFS . These servers can be used for interactive work, but some resource limits are enforced, so if you need to run a long-running or compute- and/or memory-intensive process you should submit a job. Remember, these are servers for research and academic use, not for administrative or business functions. The iron , rye and wheat servers are compute nodes. They have more CPU power and more memory than the rice servers, and are meant for both interactive jobs (where you log in to control what happens) and batch jobs (where everything runs from a script that you submit). Some wheat and iron nodes also have significantly more memory than others. Like the rice servers, these are available for use for coursework or research purposes. The oat servers are GPU compute nodes. They are similar to the compute nodes mentioned above, except that they also have GPUs installed. These nodes are meant for computational academic or research work that is able to take advantage of a GPU (e.g., TensorFlow jobs fit into this category). All FarmShare servers run Ubuntu 22.04 LTS FarmShare currently has 4 rice servers (login nodes) along with 26 compute nodes (wheat, iron, oat nodes).","title":"Cluster Components"},{"location":"#eligibility","text":"FarmShare is available to anyone who has a full-service SUNetID. A full-service SUNetID is one that has email service; if you can successfully get to Stanford Webmail , then you are eligible to use FarmShare 2 for academic work and small research jobs! If you do not already have a full-service SUNetID (maybe because you are a visiting researcher), you can get a sponsored full-service SUNetID. Read more about SUNetID levels . Note that, in order to get a sponsored SUNetID, a monthly fee will be charged by University IT. Only people with spending authority may sponsor a SUNetID. Sponsorships can be obtained and paid for through Sponsorship Manager . Current rates are available from the Sponsored Account Rates page . FarmShare is meant for low- or moderate-risk data , and is primarily intended for class work and for other general- and personal-use work (research and training). It is not meant for sponsored research (where you have a dedicated source of funding), and is not approved for handling high-risk data.","title":"Eligibility"},{"location":"#sponsored-research","text":"If you are doing sponsored or departmental research, then FarmShare might not be the right place for you. Instead, if the data you are working with is all low-risk, then you should consider getting access to our Sherlock Cluster! The Sherlock web site has more information about how to get access.","title":"Sponsored Research"},{"location":"#getting-help","text":"Most FarmShare support is provided during business hours, either via email or during academic-year office hours. For email support, send a message to srcc-support . Make sure you have \u201cFarmShare\u201d somewhere in the subject line, and please be as detailed as possible with your request.","title":"Getting Help"},{"location":"capacity/","text":"Adding Capacity \u00b6 FarmShare does not have a dedicated funding source available to it, and we appreciate any contributions that people can make. For example: If you are using FarmShare for a class, let your department chair know that you are using it. If you are a registered student group using FarmShare, let your faculty representative and/or ASSU know that you are using it. If you have spare funds available, email us and we can arrange either a iJournals transfer to our PTA, or a capital purchase from your PTA. If you already have hardware being supported by SRCC, we may be able to use that hardware once you are done with it. We appreciate anything you can do to get the word out about FarmShare and how awesome it is!","title":"Adding Capacity"},{"location":"capacity/#adding-capacity","text":"FarmShare does not have a dedicated funding source available to it, and we appreciate any contributions that people can make. For example: If you are using FarmShare for a class, let your department chair know that you are using it. If you are a registered student group using FarmShare, let your faculty representative and/or ASSU know that you are using it. If you have spare funds available, email us and we can arrange either a iJournals transfer to our PTA, or a capital purchase from your PTA. If you already have hardware being supported by SRCC, we may be able to use that hardware once you are done with it. We appreciate anything you can do to get the word out about FarmShare and how awesome it is!","title":"Adding Capacity"},{"location":"connecting/","text":"Getting Connected \u00b6 SSH on Linux and macOS \u00b6 SSH provides a secure, remote terminal connection to the login nodes. There are a number of rice servers providing login service (e.g., rice-01, rice-02); the easiest way to connect is using the load-balanced name, login.farmshare.stanford.edu , which will select one for you according to recent utilization. Simply open up your favorite terminal program (on macOS you can use the built-in Terminal application or something like iTerm) and run the ssh command like so, replacing SUNetID with your own SUNet ID: ssh SUNetID@login.farmshare.stanford.edu Depending on your environment and configuration you may be prompted for your SUNet ID password, and you\u2019ll need to complete your login using two-step authentication . SSH on Windows \u00b6 Windows does not have a built-in SSH client, but a number are available from third parties: MobaXterm \u00b6 MobaXterm is an SSH client with a built-in X server, making remote display extremely convenient, and the Home Edition is free for personal use. PuTTY \u00b6 PuTTY is a popular, freely available SSH client for Windows. The default settings are appropriate for most users, so all you need to do is specify the host name and click the Open button to connect. Aat the login as: prompt, enter your SUNet ID, and authenticate by password if asked. A number of configuration options are available, and PuTTY allows you to customize your connection settings and save them as a session. The next time you start the application you can load the saved configuration by selecting the desired session and clicking the Load button. PuTTY supports GSSAPI authentication as of version 0.61 and will automatically use your existing Kerberos credentials for login, if available. You may also want to enable Kerberos ticket forwarding by checking the Connection \u2192 SSH \u2192 Auth \u2192 GSSAPI \u2192 Allow GSSAPI credential delegation box. Mobile Shell (Mosh) \u00b6 Mosh is an alternative to SSH for Linux and macOS clients. It uses SSH for authentication, so you may want to review the suggested SSH configuration above. Mosh has some advantages, including predictive display, which can be useful on high-latency connections, and improved network resiliency. Mosh connections can persist when you switch networks and can even survive putting your computer to sleep. OnDemand \u00b6 Open OnDemand is a browser based interface to FarmShare. OnDemand offers terminal, file manager, editor, desktop, and GUI applications right from your web browser!","title":"Getting Connected"},{"location":"connecting/#getting-connected","text":"","title":"Getting Connected"},{"location":"connecting/#ssh-on-linux-and-macos","text":"SSH provides a secure, remote terminal connection to the login nodes. There are a number of rice servers providing login service (e.g., rice-01, rice-02); the easiest way to connect is using the load-balanced name, login.farmshare.stanford.edu , which will select one for you according to recent utilization. Simply open up your favorite terminal program (on macOS you can use the built-in Terminal application or something like iTerm) and run the ssh command like so, replacing SUNetID with your own SUNet ID: ssh SUNetID@login.farmshare.stanford.edu Depending on your environment and configuration you may be prompted for your SUNet ID password, and you\u2019ll need to complete your login using two-step authentication .","title":"SSH on Linux and macOS"},{"location":"connecting/#ssh-on-windows","text":"Windows does not have a built-in SSH client, but a number are available from third parties:","title":"SSH on Windows"},{"location":"connecting/#mobaxterm","text":"MobaXterm is an SSH client with a built-in X server, making remote display extremely convenient, and the Home Edition is free for personal use.","title":"MobaXterm"},{"location":"connecting/#putty","text":"PuTTY is a popular, freely available SSH client for Windows. The default settings are appropriate for most users, so all you need to do is specify the host name and click the Open button to connect. Aat the login as: prompt, enter your SUNet ID, and authenticate by password if asked. A number of configuration options are available, and PuTTY allows you to customize your connection settings and save them as a session. The next time you start the application you can load the saved configuration by selecting the desired session and clicking the Load button. PuTTY supports GSSAPI authentication as of version 0.61 and will automatically use your existing Kerberos credentials for login, if available. You may also want to enable Kerberos ticket forwarding by checking the Connection \u2192 SSH \u2192 Auth \u2192 GSSAPI \u2192 Allow GSSAPI credential delegation box.","title":"PuTTY"},{"location":"connecting/#mobile-shell-mosh","text":"Mosh is an alternative to SSH for Linux and macOS clients. It uses SSH for authentication, so you may want to review the suggested SSH configuration above. Mosh has some advantages, including predictive display, which can be useful on high-latency connections, and improved network resiliency. Mosh connections can persist when you switch networks and can even survive putting your computer to sleep.","title":"Mobile Shell (Mosh)"},{"location":"connecting/#ondemand","text":"Open OnDemand is a browser based interface to FarmShare. OnDemand offers terminal, file manager, editor, desktop, and GUI applications right from your web browser!","title":"OnDemand"},{"location":"slurm/","text":"Running Jobs \u00b6 Slurm \u00b6 FarmShare uses Slurm for job (resource) management. Full documentation and detailed usage information is provided in the man pages for the srun, sbatch, squeue, scancel, sinfo, and scontrol commands. Jobs are scheduled according to a priority which depends on a number of factors, including how long a job has been waiting, its size, and a fair-share value that tracks recent per-user utilization of cluster resources. Lower-priority jobs, and jobs requiring access to resources not currently available, may wait some time before starting to run. The scheduler may reserve resources so that pending jobs can start; while it will try to backfill these resources with smaller, shorter jobs (even those at lower priorities), this behavior can sometimes cause nodes to appear to be idle even when there are jobs that are ready to run. You can use squeue --start to get an estimate of when pending jobs will start. Interactive Jobs \u00b6 Interactive sessions that require resources in excess of limits on the login nodes, exclusive access to resources, or access to a feature not available on the login nodes (e.g., a GPU), can be submitted to a compute node. Each user is allowed one interactive job, which may run for at most one day. You can use the srun command to request one: ta5@rice-04:~$ srun --qos=interactive --pty bash ta5@wheat-01:~$ Batch Jobs \u00b6 The sbatch command is used to submit a batch job, and takes a batch script as an argument. Options are used to request specific resources (including runtime), and can be provided either on the command line or, using a special syntax, in the script file itself. sbatch can also be used to submit many similar jobs, each perhaps varying in only one or two parameters, in a single invocation using the \u2013array option; each job in an array has access to environment variables identifying its rank. Sample batch script to submit a job: ta5@rice-04:~$ cat hello_world.sh #!/bin/bash #SBATCH --job-name=hello_world #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=1 #SBATCH --partition=normal echo 'Hello World!' ta5@rice-04:~$ ta5@rice-04:~$ ta5@rice-04:~$ sbatch hello_world.sh Submitted batch job 177987 ta5@rice-04:~$ ta5@rice-04:~$ cat slurm-177987.out Hello World! ta5@rice-04:~$ Partition Info \u00b6 Partition Max Mem Max CPU normal 188GB 256 bigmem 768GB 344 normal 188GB 256 interactive 188GB 16","title":"Running Jobs"},{"location":"slurm/#running-jobs","text":"","title":"Running Jobs"},{"location":"slurm/#slurm","text":"FarmShare uses Slurm for job (resource) management. Full documentation and detailed usage information is provided in the man pages for the srun, sbatch, squeue, scancel, sinfo, and scontrol commands. Jobs are scheduled according to a priority which depends on a number of factors, including how long a job has been waiting, its size, and a fair-share value that tracks recent per-user utilization of cluster resources. Lower-priority jobs, and jobs requiring access to resources not currently available, may wait some time before starting to run. The scheduler may reserve resources so that pending jobs can start; while it will try to backfill these resources with smaller, shorter jobs (even those at lower priorities), this behavior can sometimes cause nodes to appear to be idle even when there are jobs that are ready to run. You can use squeue --start to get an estimate of when pending jobs will start.","title":"Slurm"},{"location":"slurm/#interactive-jobs","text":"Interactive sessions that require resources in excess of limits on the login nodes, exclusive access to resources, or access to a feature not available on the login nodes (e.g., a GPU), can be submitted to a compute node. Each user is allowed one interactive job, which may run for at most one day. You can use the srun command to request one: ta5@rice-04:~$ srun --qos=interactive --pty bash ta5@wheat-01:~$","title":"Interactive Jobs"},{"location":"slurm/#batch-jobs","text":"The sbatch command is used to submit a batch job, and takes a batch script as an argument. Options are used to request specific resources (including runtime), and can be provided either on the command line or, using a special syntax, in the script file itself. sbatch can also be used to submit many similar jobs, each perhaps varying in only one or two parameters, in a single invocation using the \u2013array option; each job in an array has access to environment variables identifying its rank. Sample batch script to submit a job: ta5@rice-04:~$ cat hello_world.sh #!/bin/bash #SBATCH --job-name=hello_world #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=1 #SBATCH --partition=normal echo 'Hello World!' ta5@rice-04:~$ ta5@rice-04:~$ ta5@rice-04:~$ sbatch hello_world.sh Submitted batch job 177987 ta5@rice-04:~$ ta5@rice-04:~$ cat slurm-177987.out Hello World! ta5@rice-04:~$","title":"Batch Jobs"},{"location":"slurm/#partition-info","text":"Partition Max Mem Max CPU normal 188GB 256 bigmem 768GB 344 normal 188GB 256 interactive 188GB 16","title":"Partition Info"},{"location":"software/","text":"Using Software \u00b6 FarmShare has lots of free and commercial software available to use. We provide software that comes with Ubuntu, software that we package ourselves, and we also provide the capability for you to build and use software yourself. Software Sources \u00b6 Software on FarmShare comes from three sources: Packages Modules Build your own Packages \u00b6 Packaged software is easiest to use, because you don\u2019t have to do anything. Packaged software has already been installed on all of the systems in the environment, so to use the software, you just have to run the command. For example, to run the packaged version of Python, you just need to run the python3 command: ta5@rice-01:~$ python3 --version Python 3.10.12 FarmShare 2 runs Ubuntu 22.04 LTS, which means almost anything in the Ubuntu Jammy package repository is available to be installed. Modules \u00b6 To provide up-to-date software, we use modules. Modularized software lives in shared, network-connected storage, and is built by the FarmShare support team. To see a listing of available software, run the module available command: ta5@rice-01:~$ module available ---------------------------- /software/modules/linux-ubuntu22.04-x86_64/Core ---------------------------- apptainer/1.1.9 libpng/1.5.30 blast-plus/2.14.1 llvm/16.0.6 boost/1.83.0 micromamba/1.1.0 bowtie2/2.5.1 mpich/4.1.2 cuda/11.4.4 numactl/2.0.14 cudnn/8.2.4.15-11.4 openblas/0.3.23 gcc/13.2.0 openblas/0.3.24 (D) imagemagick/7.1.1-11 openmpi/1.10.7 intel-oneapi-compilers/2023.2.1 py-jupyterlab/4.0.1_python/3.11.6 intel-oneapi-dal/2023.2.0 py-pip/23.1.2_python/3.11.6 intel-oneapi-ipp/2021.9.0 python/3.11.6 intel-oneapi-mkl/2023.2.0 r-magick/2.7.4_r/4.3.0 intel-oneapi-mpi/2021.10.0 r-tidyverse/2.0.0_r/4.3.0 intel-oneapi-tbb/2021.10.0 r/4.3.0 julia/1.9.2 rust/1.75.0 libjpeg-turbo/2.1.5.1 ------------------------------------- /software/modules/commercial -------------------------------------- mathematica/13.3.1 mathematica/14.0.0 (D) matlab/r2023b sas/9.4 stata/18 Where: D: Default Module If the avail list is too long consider trying: \"module --default avail\" or \"ml -d av\" to just list the default modules. \"module overview\" or \"ml ov\" to display the number of modules for each name. Use \"module spider\" to find all possible modules and extensions. Use \"module keyword key1 key2 ...\" to search for all possible modules matching any of the \"keys\". Build Your Own \u00b6 In addition to the software that we provide for you to use, it is perfectly OK for you to build and use your own software.","title":"Using Software"},{"location":"software/#using-software","text":"FarmShare has lots of free and commercial software available to use. We provide software that comes with Ubuntu, software that we package ourselves, and we also provide the capability for you to build and use software yourself.","title":"Using Software"},{"location":"software/#software-sources","text":"Software on FarmShare comes from three sources: Packages Modules Build your own","title":"Software Sources"},{"location":"software/#packages","text":"Packaged software is easiest to use, because you don\u2019t have to do anything. Packaged software has already been installed on all of the systems in the environment, so to use the software, you just have to run the command. For example, to run the packaged version of Python, you just need to run the python3 command: ta5@rice-01:~$ python3 --version Python 3.10.12 FarmShare 2 runs Ubuntu 22.04 LTS, which means almost anything in the Ubuntu Jammy package repository is available to be installed.","title":"Packages"},{"location":"software/#modules","text":"To provide up-to-date software, we use modules. Modularized software lives in shared, network-connected storage, and is built by the FarmShare support team. To see a listing of available software, run the module available command: ta5@rice-01:~$ module available ---------------------------- /software/modules/linux-ubuntu22.04-x86_64/Core ---------------------------- apptainer/1.1.9 libpng/1.5.30 blast-plus/2.14.1 llvm/16.0.6 boost/1.83.0 micromamba/1.1.0 bowtie2/2.5.1 mpich/4.1.2 cuda/11.4.4 numactl/2.0.14 cudnn/8.2.4.15-11.4 openblas/0.3.23 gcc/13.2.0 openblas/0.3.24 (D) imagemagick/7.1.1-11 openmpi/1.10.7 intel-oneapi-compilers/2023.2.1 py-jupyterlab/4.0.1_python/3.11.6 intel-oneapi-dal/2023.2.0 py-pip/23.1.2_python/3.11.6 intel-oneapi-ipp/2021.9.0 python/3.11.6 intel-oneapi-mkl/2023.2.0 r-magick/2.7.4_r/4.3.0 intel-oneapi-mpi/2021.10.0 r-tidyverse/2.0.0_r/4.3.0 intel-oneapi-tbb/2021.10.0 r/4.3.0 julia/1.9.2 rust/1.75.0 libjpeg-turbo/2.1.5.1 ------------------------------------- /software/modules/commercial -------------------------------------- mathematica/13.3.1 mathematica/14.0.0 (D) matlab/r2023b sas/9.4 stata/18 Where: D: Default Module If the avail list is too long consider trying: \"module --default avail\" or \"ml -d av\" to just list the default modules. \"module overview\" or \"ml ov\" to display the number of modules for each name. Use \"module spider\" to find all possible modules and extensions. Use \"module keyword key1 key2 ...\" to search for all possible modules matching any of the \"keys\".","title":"Modules"},{"location":"software/#build-your-own","text":"In addition to the software that we provide for you to use, it is perfectly OK for you to build and use your own software.","title":"Build Your Own"}]} \ No newline at end of file diff --git a/sitemap.xml.gz b/sitemap.xml.gz index 7e3c77b..6a90c92 100644 Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ diff --git a/slurm/index.html b/slurm/index.html index 056ee72..e3478f6 100644 --- a/slurm/index.html +++ b/slurm/index.html @@ -143,18 +143,31 @@

Partition InfoPartition +Max Mem +Max CPU -normal -188GB -256 -2 Days +normal +188GB +256 + + +bigmem +768GB +344 + + +normal +188GB +256 + + +interactive +188GB +16