https://swsmith.cc/ Recent content in Hi, I'm Sean 🚀 on Sean Smith 🚀 Hugo -- gohugo.io en-us Mon, 23 Sep 2024 00:00:00 +0000 https://swsmith.cc/posts/mt-blanc.html Mon, 23 Sep 2024 00:00:00 +0000 https://swsmith.cc/posts/mt-blanc.html Imagine you show up in Chamonix, France with no gear and a desire to climb the highest peak in Europe. You have the skills from previous glacier climbs like Mt Hood and Mt. Baker but you don’t have your alpine climbing gear. That’s where we found ourselves at the start of September 2024. In three weeks we managed to buy/rent all the gear, setup accommodations in the mountain huts and summit on September 22nd without a guide. https://swsmith.cc/posts/starlink.html Sun, 01 Sep 2024 00:00:00 +0000 https://swsmith.cc/posts/starlink.html We recently installed Starlink mini as a permanent installation in our van. Here’s what we learned: What is Starlink? Starlink is a satellite internet constellation operated by SpaceX. It provides internet access to wherever we are in the van both in motion and stationary. In rural Alaska we often don’t have cell service at all and so installing the Starlink dish on the van was vital for maintaining connectivity. We’ve used the Generation 2 before and the Generation 3 dish but didn’t install either permanently. https://swsmith.cc/posts/fly-to-alaska.html Mon, 17 Jun 2024 00:00:00 +0000 https://swsmith.cc/posts/fly-to-alaska.html How to fly to Alaska This is a story and hopefully some useful insights on those looking to fly up from the lower 48 to Alaska in their own airplane. Most of the information I gained from Ted Waltman’s excellent website https://fly2ak.com/, so check that out first. We flew up from Seattle Washington to Anchorage over the course of 5 days and took the Alcan highway route, our route of flight was: https://swsmith.cc/posts/gpu-memory.html Tue, 21 May 2024 00:00:00 +0000 https://swsmith.cc/posts/gpu-memory.html GPU Memory needs scale up based on the size of the model. To quickly calculate the memory required for a model you can use the calculators below. For inference memory required is typically 2 x the number of parameters, this is because each parameter is typically two bytes (FP16). So a 7B parameter model takes 14GB. For training in mixed precision (FP16) it’s typically 18x the number of parameters plus activations, this is typically 22-32x the number of parameters. https://swsmith.cc/posts/ec2-topology.html Mon, 08 Jan 2024 00:00:00 +0000 https://swsmith.cc/posts/ec2-topology.html Click for Demo! What is the EC2 Topology API? AWS Instance Topology gives you details on the interconnect between ec2 instances. It’s helpful to determine if all the nodes are co-located on the same spine and can even be used when trying to make scaling decision for large ML training runs or HPC jobs. To call the API, you’ll need at least aws cli version 2.13.36. You can check the version with: https://swsmith.cc/posts/efa-best-practices.html Tue, 12 Dec 2023 00:00:00 +0000 https://swsmith.cc/posts/efa-best-practices.html Elastic Fabric Adaptor (EFA) is is like a race car, it enables super-fast, os-bypass, high speed networking when working properly but quickly breaks down (aka falls back to TCP) when it isn’t configured properly. The following guide is some best practices when working with the EFA drawn on my 6 years of experience working with it. First some basics: What is EFA and how is it different than Infiniband? https://swsmith.cc/posts/grafana-slurm.html Thu, 26 Oct 2023 00:00:00 +0000 https://swsmith.cc/posts/grafana-slurm.html Grafana is an open source tool that allows us to create dashboards and monitor our cluster. In the following guide we’ll show you how to setup Grafana, Prometheus, Slurm exporter and DCGM Exporter to monitor a cluster. This will help you answer questions like: how many jobs/instances are running CPU utilization GPU Utilization Memory usage EFA (Network) Traffic Disk iops We’ll setup the following exporters but don’t limit yourself to just these. https://swsmith.cc/posts/containers-aws-parallelcluster.html Sun, 13 Aug 2023 00:00:00 +0000 https://swsmith.cc/posts/containers-aws-parallelcluster.html What are containers? Containers are a great way to package software, they wrap the runtime of the software up with the application’s code. This allows you to pull down optimized software containers and run them out of the box without all the complications of compiling them for a new system. In this blog we’ll focus on the nvidia container repository (ngc) since they have optimized containers for applications like gromacs, nemo, and BERT. https://swsmith.cc/posts/setup-email-notifications.html Sun, 13 Aug 2023 00:00:00 +0000 https://swsmith.cc/posts/setup-email-notifications.html Update (5/23/24): the author of this plugin, Neil Munday published a recipe on AWS HPC Recipes. Please see that for the most up to date guidance. Slurm has an option to send emails when your job changes status. This is useful to get notifications when your job completes. For example in my sbatch script I could add: #SBATCH --mail-user=sean@swsmith.cc After the job completes, you’ll get an email: You can include useful information such as stdout, stderr, runtime ect. https://swsmith.cc/posts/slurm-pam-adopt.html Fri, 04 Aug 2023 00:00:00 +0000 https://swsmith.cc/posts/slurm-pam-adopt.html Slurm PAM “adopt” allows you to restrict users to login to only to compute nodes where they have running jobs. It “adopts” the process that the user is running on the node in order to login. If they don’t have a job running they’ll get permission denied. There is some nuance to this, for example should users be allowed to ssh into a node they’re only partially using? This behavior can be configured with Options in the command. https://swsmith.cc/posts/link-iam-to-ad.html Thu, 03 Aug 2023 00:00:00 +0000 https://swsmith.cc/posts/link-iam-to-ad.html In this blogpost we’ll describe how to manager POSIX user identities through IAM identity center by setting up propagation to Active Directory. This can be used to create user identities on your HPC cluster without going through the pain of creating them in Active Directory (which requires windows). This also allows you to link the IAM user with their POSIX user and give users a 1-click login onto these instances using SSM RunAsUser support. https://swsmith.cc/posts/slurm-multi-cluster.html Mon, 31 Jul 2023 00:00:00 +0000 https://swsmith.cc/posts/slurm-multi-cluster.html Slurm supports a feature called multi-cluster mode this allows you to submit jobs across multiple clusters, for example you could have one cluster on-prem, another in the cloud, or you could have clusters in two different regions, or you could have an x86 cluster and an aarch64 one. These are all scenarios where it might make sense to have two separate clusters, both running an independent slurmctld processes but which allow users to submit between the different clusters. https://swsmith.cc/posts/hpc7g-parallelcluster.html Mon, 10 Jul 2023 00:00:00 +0000 https://swsmith.cc/posts/hpc7g-parallelcluster.html HPC7g instances are the first ARM based HPC instances in AWS. These instances combine excellent per-core pricing, deep capacity pools and 200 GB EFA networking in order to create the perfect HPC instance for large-scale cost-effective simulations. There’s three different sizes: Instance Size Cores Memory (GiB) EFA Network Bandwidth Price (On-Demand in us-east-1) hpc7g.4xlarge 16 128 200 GBps 1.683 hpc7g.8xlarge 32 128 200 GBps 1. https://swsmith.cc/posts/pilot.html Thu, 08 Jun 2023 00:00:00 +0000 https://swsmith.cc/posts/pilot.html Flying is the most rewarding skills you can acquire; it’s the perfect mix between knowledge and skill. You have to be able to make decisions quickly and think on your feet, in addition to being able to study complex systems, understand airspace and complex approaches. The U.S. has a phenomenal system for general aviation. As a private pilot you can basically fly anywhere, during the day, at night, through the weather (with an additional rating). https://swsmith.cc/posts/fsx-openzfs.html Wed, 24 May 2023 00:00:00 +0000 https://swsmith.cc/posts/fsx-openzfs.html FSx OpenZFS is a new filesystem offering that provides a managed OpenZFS filesystem. In previous blogposts we’ve showed how to use FSx Lustre, FSx Netapp Ontap and EFS with AWS ParallelCluster. In this blogpost we’ll show you how to create and mount OpenZFS filesystems on ParallelCluster. Before we start, when should you use OpenZFS? So when should you use FSx OpenZFS? NFS compliant filesystem Fast filesystem performance for 30% cheaper than FSx Lustre Built in support for backups Multi-AZ support So when shouldn’t you use FSx OpenZFS? https://swsmith.cc/posts/hugo-amplify.html Wed, 10 May 2023 00:00:00 +0000 https://swsmith.cc/posts/hugo-amplify.html It should be no secret that I ♥️ Hugo and Amplify. Combined I’ve built out nearly a dozen websites using these tools, including hpcworkshops.com, pcluster.cloud, thefiftyproject.com and of course this website. These websites are hosted on AWS Amplify for a grand total of $.30/month with no server maintenance ever needed. Deployments are done using Github actions and new changes are automatically built and published when new commits are pushed to Github. https://swsmith.cc/posts/flexlm.html Wed, 19 Apr 2023 00:00:00 +0000 https://swsmith.cc/posts/flexlm.html FlexLM is the most popular license server for HPC workloads. It’s the license server for applications like Siemens StarCCM+, Ansys Fluent, Abaqus, ect. In the following guide we’ll setup a license server that can be used with FlexLM (or any other linux-based licensing client). Once you have the licensing setup you can setup license tracking in Slurm following the blogpost Setup Licensing with AWS ParallelCluster and Slurm 🪪. Instance Type FlexLM has pretty minor compute requirements, just 2 cores and 4 GB of memory, so it can be easily run on a small (cheap) instance like the t3. https://swsmith.cc/posts/cost-explorer-pcluster.html Thu, 06 Apr 2023 00:00:00 +0000 https://swsmith.cc/posts/cost-explorer-pcluster.html Cost Explorer allows you to track cost at the cluster, queue, user and job level. It does this by tracking tags applied automatically to EC2 Instances launched with parallelcluster. This gives users a view of exactly how much a cluster costs overtime, it tracks: EC2 Instances EBS Volumes FSx Lustre Volumes EFS Volumes Data Transfer Setup Ok so how do we set this up? Activate the parallelcluster:cluster-name tag in the Billing Dashboard > Cost Allocation Tags Update: as of June 8th 2022, you can activate these tags programmatically from the AWS API, CLI or SDK. https://swsmith.cc/posts/s3-mountpoint-aws-pcluster.html Fri, 24 Mar 2023 00:00:00 +0000 https://swsmith.cc/posts/s3-mountpoint-aws-pcluster.html S3 Mountpoint is a new solution that provides a way to mount an S3 bucket as if it’s a shared filesystem. This gives several advantages: Allows using applications that don’t support S3 API’s Easy interface for users to browse files in S3 So what are the downsides? Read-only in alpha GA version released March 2023 Non-POSIX compliant. This means operations such as metadata changes, file-locking ect. are not supported. https://swsmith.cc/posts/slurm-rest-api.html Wed, 08 Mar 2023 00:00:00 +0000 https://swsmith.cc/posts/slurm-rest-api.html The Slurm REST API can be used to programmatically submit and monitor jobs on the cluster. Step 1 - Setup Slurm Accounting First setup Slurm Accounting following the instructions. Step 2 - Create a Security Group to allow inbound API Requests By default, your cluster will not be able to accept incoming HTTPS requests to the REST API. You will need to create a security group to allow traffic from outside the cluster to call the API. https://swsmith.cc/posts/fsx-lustre-cache.html Tue, 28 Feb 2023 00:00:00 +0000 https://swsmith.cc/posts/fsx-lustre-cache.html Overview FSx Lustre is a powerful filesystem for workloads that require low-latency, parallel access, however this comes at a cost, sometimes 5 x greater than S3. To avoid paying to keep all your data in Lustre, you can setup a link between S3 and FSx Lustre. When data is requested it’s pulled from S3 and “cached” in Lustre and when it’s no longer needed it’s evicted from Lustre and the only copy sits in S3. https://swsmith.cc/posts/aviation-led-sectional.html Tue, 14 Feb 2023 00:00:00 +0000 https://swsmith.cc/posts/aviation-led-sectional.html This blog is typically about AWS and HPC however once and a while I get to talk about other stuff like my 50 States Project or this Aviation Sectional I built. The idea was to build a map, known in aviation parlance as a sectional, that illuminated airports based on the current weather condition. In aviation there’s different flight rules that apply when conditions are below certain thresholds. I’ll save you all the details but just know green = good, blue is less so, red is more work and purple means you probably shouldn’t be flying a small aircraft. https://swsmith.cc/posts/slurm-login-aws-parallelcluster.html Fri, 20 Jan 2023 00:00:00 +0000 https://swsmith.cc/posts/slurm-login-aws-parallelcluster.html Update: This has been written up on the ParallelCluster Wiki: ParallelCluster: Launching a Login Node Some reasons why you may want to use a Login Node: Separation of scheduler slurmctld process from users. This helps prevent a case where a user consumes all the system resources and Slurm can no longer function. Ability to set different IAM permissions for Login versus Head Node. I’ve divided the setup into two parts: https://swsmith.cc/posts/spack.html Wed, 07 Dec 2022 00:00:00 +0000 https://swsmith.cc/posts/spack.html Install Spack Spack is a package manager for supercomputers, Linux, and macOS. It makes installing scientific software easy. Spack isn’t tied to a particular language; you can build a software stack in Python or R, link to libraries written in C, C++, or Fortran, and easily swap compilers or target specific microarchitectures. First, on the head node - which we connected to via SSM or DCV we’ll run: export SPACK_ROOT=/shared/spack git clone -b v0. https://swsmith.cc/posts/starccm-aws-parallelcluster.html Sat, 03 Dec 2022 00:00:00 +0000 https://swsmith.cc/posts/starccm-aws-parallelcluster.html Siemens StarCCM+ is a multiphysics computational fluid dynamics (CFD) software for the simulation of products operating under real-world conditions. Simcenter STAR-CCM+ uniquely brings automated design exploration and optimization to the CFD simulation toolkit of every engineer. In this blogpost we’ll show you how to setup StarCCM+ using AWS ParallelCluster, which will allow you to scale out your CFD simulations to thousands of cores. Step 1: Setup a Cluster I’ll also assume you have AWS ParallelCluster Manager setup, if you don’t follow the instructions on hpcworkshops. https://swsmith.cc/posts/dcv-visualization-queue.html Tue, 29 Nov 2022 00:00:00 +0000 https://swsmith.cc/posts/dcv-visualization-queue.html Note: this has been turned into an AWS Blog! 🚀 Check it out: Elastic DCV Queues with NICE DCV DCV Visualization Queue When DCV is enabled, the default behaviour of AWS ParallelCluster is to run a single DCV session on the head node, this is a quick and easy way to visualize the results of your simulations or run a desktop application such as StarCCM+. A common ask is to run DCV sessions on a compute queue instead of the head node. https://swsmith.cc/posts/ebs-pcluster.html Tue, 08 Nov 2022 00:00:00 +0000 https://swsmith.cc/posts/ebs-pcluster.html In previous blogposts we looked at several approaches to add shared storage to a cluster, these all focus on mounting a shared filesystem. If you have Multi-Node (MPI) style jobs, this is likely a requirement. You can follow those guides below: Mount FSx Netapp ONTAP with AWS ParallelCluster Setup FSx Lustre PERSISTENT_2 with AWS ParallelCluster Mount Additional EFS/FSx Lustre Filesystems in AWS ParallelCluster Let’s say you don’t need shared storage but rather local storage on each compute node. https://swsmith.cc/posts/multi-az-parallelcluster.html Wed, 02 Nov 2022 00:00:00 +0000 https://swsmith.cc/posts/multi-az-parallelcluster.html Today we launched a new version of AWS ParalleCluster, version 3.3.0. This version has a beta feature hidden in the release log: Allow for suppressing the SingleSubnetValidator for Queues. With this feature, we can setup a single AZ-per queue essentially allowing us to choose which Availibility Zone is associated with each queue. This is useful for capacity constrained instances, such as GPU and HPC instances which may exist in different availibility zones. https://swsmith.cc/posts/fsx-filecache-pcluster.html Mon, 24 Oct 2022 00:00:00 +0000 https://swsmith.cc/posts/fsx-filecache-pcluster.html Amazon File Cache is a new service that provides a cache to use on-prem data in the cloud but it has a few advantages over SCP/SFTP and Datasync. Single namespace - files & metadata are copied up and down transparently to the user Support for S3 and NFSv3 (Not NFSv4 as of this writing) Lazy Loading - files are pulled in as needed, resulting in a smaller overall cache size So when should you use File Cache? https://swsmith.cc/posts/ls-dyna-aws-parallelcluster.html Wed, 19 Oct 2022 00:00:00 +0000 https://swsmith.cc/posts/ls-dyna-aws-parallelcluster.html LS-Dyna is a popular Finite Element Analysis (FEA) Simulation software that uses explicit time integration, it’s commonly used in automotive to simulate crashes but can simulate any type of impact. It was originally developed at Lawrence Livermore National Lab then spun out into it’s own company (LSTC) before Ansys bought the company in 2019. Many of the artifacts from the LSTC days still exist, such as the LSTC license manager, ftp site ect. https://swsmith.cc/posts/ansys-lstc-license-manager.html Tue, 18 Oct 2022 00:00:00 +0000 https://swsmith.cc/posts/ansys-lstc-license-manager.html LSTC License Manager Instructions on how to download and setup the LS-Dyna License server from Ansys. Make sure you have the login credentials to the FTP site, i.e. you can access: https://ftp.lstc.com/user/license/License-Manager/LSTC_LicenseManager-InstallationGuide.pdf. If you don’t, contact Ansys. Note: In this guide I’m showing an example of how to setup a dedicated license server, however these steps can also be done on the HeadNode of a cluster. I reccomend keeping a dedicated license server so you can upgrade the cluster overtime, however for one-off cases it might be make sense to setup the license locally, if so skip to step 2. https://swsmith.cc/posts/slurm-constraint.html Fri, 23 Sep 2022 00:00:00 +0000 https://swsmith.cc/posts/slurm-constraint.html In previous posts we discussed adding multiple instances to same Slurm queue and enabling Fast Failover, this is great when you’re flexible on the specific instance type used to run your job, but what if you want to choose the instance type at job submission time? In this blogpost we look at how to use the Slurm --constraint flag to pick the specific instance type at runtime. Setup To setup, refer to the Fast Failover Setup section. https://swsmith.cc/posts/external-home-pcluster.html Tue, 30 Aug 2022 00:00:00 +0000 https://swsmith.cc/posts/external-home-pcluster.html External filesystems can be mounted and used as home directories in AWS ParallelCluster. This has several advantages over the default, which is an EBS volume on the head node /home shared via NFSv4 to the compute nodes. Home directories can be persisted after cluster deletion, saving data and allowing users to reproduce the same environment Home directories can be mounted on multiple clusters, allowing users to have the same filesystem between different clusters Reduce dependency on HeadNode, this allows you to size down the HeadNode since it’s no longer serving critical traffic i. https://swsmith.cc/posts/gromacs-pcluster.html Tue, 16 Aug 2022 00:00:00 +0000 https://swsmith.cc/posts/gromacs-pcluster.html Note: this has been turned into an AWS Workshop! 🚀 Check it out: Gromacs on AWS ParallelCluster Gromacs is a popular open source Molecular Dynamics application. It supports GPU and CPU acceleration and supports multi-node processing using MPI. In the following guide we’ll setup a MPI compatible version of Gromacs using Spack package manager. Setup In this guide, I’ll assume you already have AWS ParallelCluster Manager setup, if you don’t follow the instructions on hpcworkshops. https://swsmith.cc/posts/spot-failover-ondemand.html Thu, 14 Jul 2022 00:00:00 +0000 https://swsmith.cc/posts/spot-failover-ondemand.html Spot pricing, up to 90% off On-Demand, is enticing but comes with the caveat that your jobs can get terminated with only a 2-minute warning. So what happens when your instance gets a Spot termination and you need to finish that job? In the following blogpost I present a solution that’ll re-run failed Spot jobs on On-Demand. If you combine this with a benchmarking solution such as Gromacs Checkpointing or StarCCM+ Save State you can resume from the last checkpoint. https://swsmith.cc/posts/fast-failover.html Mon, 11 Jul 2022 00:00:00 +0000 https://swsmith.cc/posts/fast-failover.html In a previous blogpost I showed a method of doing all-or-nothing scaling using AWS ParallelCluster. This is a great way to save on costs by making sure your job either runs or doesn’t, but what happens when your job can’t execute due to capacity? In this blog I present a method for re-trying the job quickly on other similar instance types, allowing you to diversify capacity while maintaining the same number of cores-per-instance. https://swsmith.cc/posts/starccm-save-state.html Wed, 06 Jul 2022 00:00:00 +0000 https://swsmith.cc/posts/starccm-save-state.html Spot termination gives a 2-minute warning before terminating the instance. This time period allows you to gracefully save data in order to resume later. In the following I describe how this can be done with StarCCM+ in AWS ParallelCluster 3.X however this can be generalized to any application that has the ability ot checkpoint and save current state. An example of this is Gromacs, for which I wrote a blogpost on how to enable checkpointing. https://swsmith.cc/posts/parallelcluster-multi-user.html Mon, 27 Jun 2022 00:00:00 +0000 https://swsmith.cc/posts/parallelcluster-multi-user.html See #170 This approach is a fairly lightweight way of adding users, it gives all users the same permissions. If you want a more robust multi-user approach I suggest you setup AD authentication. In order to create a user for the cluster, that user needs to exist on all the compute nodes. If they don’t Slurm won’t be able to schedule jobs and you won’t be able to run mpi jobs across multiple nodes. https://swsmith.cc/posts/all-or-nothing.html Tue, 21 Jun 2022 00:00:00 +0000 https://swsmith.cc/posts/all-or-nothing.html Update: This has been turned into an official AWS Blogpost: Minimize HPC compute costs with all-or-nothing instance launching Enable All-or-Nothing Scaling with AWS ParallelCluster All or nothing scaling is useful when you need to run MPI jobs that can’t start until all N instances have joined the cluster. The way Slurm launches instances is in a best-effort fashion, i.e. if you request 10 instances but it can only get 9, it’ll provision 9 then keep trying to get the last instance. https://swsmith.cc/posts/slurm-accounting-aws-parallelcluster.html Tue, 24 May 2022 00:00:00 +0000 https://swsmith.cc/posts/slurm-accounting-aws-parallelcluster.html In this tutorial we will work through setting up Slurm Accounting. This enables many features within slurm, including job resource tracking and providing a necessary building block to slurm federation. Step 1 - Setup External Accounting Database The first requirement is to setup an external database that Slurm can use to store the accounting data. Use the following CloudFormation Quick-Create link to create the database in your AWS account. https://swsmith.cc/posts/aws-parallelcluster-dynamic-fsxl.html Fri, 20 May 2022 00:00:00 +0000 https://swsmith.cc/posts/aws-parallelcluster-dynamic-fsxl.html You can dynamically create a filesystem per-job, this is useful for jobs that require a fast filesystem but don’t want to pay to have the filesystem running 24/7. It’s also useful to create a filesystem per-job to make sure that job has the fastest possible throughput. In order to accomplish this without wasting time waiting for the filesystem to create (~10 mins), we’ve seperated this into three seperate jobs: Create filesystem, only needs a single EC2 instance to run, can be run on head node. https://swsmith.cc/posts/fsxn-pcluster.html Mon, 16 May 2022 00:00:00 +0000 https://swsmith.cc/posts/fsxn-pcluster.html FSx Netapp ONTAP is a multi-protocol filesystem. It mounts on Windows as SMB, Linux as NFS and Mac. This allows cluster users to bridge their Windows and Linux machines with the same filesystem, potentially running both windows and linux machines for a post-processing workflow. Since 3.2.0, FSx Netapp is a supported filesystem type in AWS ParallelCluster, this means you can mount the filesystem directly through the config without having to specify a post-install script. https://swsmith.cc/posts/disable-hpc6a-cores.html Mon, 09 May 2022 00:00:00 +0000 https://swsmith.cc/posts/disable-hpc6a-cores.html How to disable hpc6a.48xlarge cores Due to the EPYC architecture, it makes more sense to disable specific cores rather than let the scheduler choose which cores to run on. This is because each ZEN 3 core is attached to a compute complex that’s made up of 4 cores, L2 and L3 cache, by disabling 1, 2 or 3 cores from the same compute complex, we increase the memory bandwidth of the remaining cores. https://swsmith.cc/posts/aws-batch-fsxlustre.html Sat, 30 Apr 2022 00:00:00 +0000 https://swsmith.cc/posts/aws-batch-fsxlustre.html Mount FSx Lustre on AWS Batch This guide describes how to mount FSx Lustre filesystem. I give an example cloudformation stack to create the AWS Batch resources. I loosely follow this guide. For the parameters, it’s important that the Subnet, Security Group, FSx ID and Fsx Mount Name follow the guidelines below: Parameter Description Subnet ID I suggest launching the batch job in the same subnet as the Security Group Must allow mounting the filesystem port 988. https://swsmith.cc/posts/slurm-failover-ondemand.html Thu, 28 Apr 2022 00:00:00 +0000 https://swsmith.cc/posts/slurm-failover-ondemand.html Slurm Failover from Spot to On-Demand In AWS ParallelCluster you can setup a cluster with two queues, one for Spot pricing and one for On-demand. When a job fails, due to a spot reclaimation, you can automatically requeue that job to OnDemand. To set that up, first create a cluster with a Spot and OnDemand queue: - Name: od ComputeResources: - Name: c6i-od-c6i32xlarge MinCount: 0 MaxCount: 4 InstanceType: c6i.32xlarge Efa: Enabled: true GdrSupport: true DisableSimultaneousMultithreading: true Networking: SubnetIds: - subnet-846f1aff PlacementGroup: Enabled: true - Name: spot ComputeResources: - Name: c6i-spot-c6i32xlarge MaxCount: 4 InstanceType: c6i. https://swsmith.cc/posts/fsx-persistent-2-pcluster.html Sat, 19 Feb 2022 00:00:00 +0000 https://swsmith.cc/posts/fsx-persistent-2-pcluster.html Overview AWS ParallelCluster only supports PERSISTENT_1, SCRATCH_1 and SCRATCH_2 as filesystems created by the cluster, however to launch filesystems with PERSISTENT_2 (announced at re:Invent 2021), you can create the filesystem outside of pcluster and then mount in the config. Why use PERSISTENT_2? 40% cheaper for the same throughput. See AWS FSx Lustre Pricing Link multiple S3 Buckets with the same Filesystem Link and de-link buckets after filesystem creation Setup From the AWS ParallelCluster docs we learn: https://swsmith.cc/posts/ec2-instance-connect.html Fri, 11 Feb 2022 00:00:00 +0000 https://swsmith.cc/posts/ec2-instance-connect.html Connect to AWS ParallelCluster with EC2 Instance Connect EC2 Instance connect allows you to SSH into an EC2 instance without a keypair. You can also perform basic file transfer i.e. SFTP and SCP with it. Advantages: doesn’t require an SSH keypair connects to private IP addresses (you still need network connectivity) You can read more about it here. First install the ec2instanceconnectcli helper: pip install ec2instanceconnectcli Connect to the instance with the mssh command: export AWS_DEFAULT_REGION='us-east-2' mssh $(pcluster describe-cluster --cluster-name hpc6a | jq -r '. https://swsmith.cc/posts/copy-ami-cross-account.html Wed, 15 Dec 2021 00:00:00 +0000 https://swsmith.cc/posts/copy-ami-cross-account.html Copy AMI cross-account This method works to copy any AMI from one account to another (without sharing or making it a public ami), or across different partitions, i.e. Commercial to GovCloud. In Commercial transfer AMI from ec2 to s3 aws ec2 create-store-image-task \ --image-id ami-1234567890abcdef0 \ --bucket myamibucket Wait for the transfer to complete, you can monitor it’s progress: aws ec2 describe-store-image-tasks Copy the AMI to an S3 bucket in Govcloud aws s3 cp s3://myamibucket/ami-1234567890abcdef0. https://swsmith.cc/posts/slurm-license-accounting.html Fri, 03 Dec 2021 00:00:00 +0000 https://swsmith.cc/posts/slurm-license-accounting.html Setup Licensing with AWS ParallelCluster and Slurm Slurm has the ability to track licenses, for example if you have 100 LS-Dyna licenses available, you can have jobs that would exceed that amount stay in pending until some of the licenses free up. Slurm has two ways of doing this: Local Licenses - Local licenses are local to the cluster in the slurm.conf. Use this if you have only one cluster. https://swsmith.cc/posts/soca-custom-ami.html Fri, 29 Oct 2021 00:00:00 +0000 https://swsmith.cc/posts/soca-custom-ami.html Import custom AMI to provision capacity faster By default, SOCA provision a vanilla AMI and install all required packages in ~3 to 5 minutes. If this cold time is not acceptable for your workload, you can launch AlwaysOn instance or pre-bake your AMI with all required libraries. Step 1: Locate your base AMI Run cat /etc/environment | grep SOCA_INSTALL_AMI on your scheduler host $ ssh -i <key> ec2-user@<ip> Last login: Wed Oct 2 20:06:47 2019 from <ip> _____ ____ ______ ___ / ___/ / __ \ / ____// | \__ \ / / / // / / /| | ___/ // /_/ // /___ / ___ | /____/ \____/ \____//_/ |_| Cluster: soca-uiupdates > source /etc/environment to SOCA paths [ec2-user@ip-30-0-1-28 ~]$ cat /etc/environment | grep SOCA_INSTALL_AMI export SOCA_INSTALL_AMI=ami-082b5a644766e0e6f [ec2-user@ip-30-0-1-28 ~]$ Step 2: Launch a temporary EC2 instance Launch a new EC2 instance using the SOCA_INSTALL_AMI image Step 3: Pre-configure your AMI ! https://swsmith.cc/posts/aws-parallelcluster-multi-fs.html Wed, 22 Sep 2021 00:00:00 +0000 https://swsmith.cc/posts/aws-parallelcluster-multi-fs.html Mount Additional EFS Filesystems in AWS ParallelCluster In AWS ParallelCluster 3.0 only one EFS filesystem can be mounted at a time. This guide allows you to attach multiple by making use of the Custom Bootstrap Actions feature to create a OnNodeConfigured script that mounts the Filesystem. To create the mount script we’ll match the options that parallelcluster uses when it launches a filesystem. See efs_mount.rb for more info. First create a script efs. https://swsmith.cc/posts/github-best-practices.html Mon, 14 Jun 2021 00:00:00 +0000 https://swsmith.cc/posts/github-best-practices.html tl;dr One commit, one feature! Specific commit messages No merge commits! 0. Supercharge GIT 🦸‍♂️ Create a global ~/.gitconfig file and include the following (change my name, email and home dir obviously): [alias] st = status ci = commit br = branch co = checkout lg = "log --color --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit --" [user] email = email@domain.com name = Sean Smith [rebase] autoStash = true [core] excludesfile = /Users/username/. https://swsmith.cc/posts/ansys-rsm-pcluster.html Wed, 09 Dec 2020 00:00:00 +0000 https://swsmith.cc/posts/ansys-rsm-pcluster.html Ansys Remote Solve Manager SOCA Remote Solve Manager is an Ansys software which enable PBS job submission from Ansys WorkBench interface. First install First set your Ansys Root, we’ll use this throughout the doc to edit files: export ANSYS_ROOT=/apps/ansys_inc/v202 ll $ANSYS_ROOT # make sure it exists Note Ansys switched from Python 2 to 3 between releases 19.2 and 20.2, the following scripts only work in Python 3. To make this work for versions < 19. https://swsmith.cc/posts/intel-select-solution.html Wed, 04 Nov 2020 00:00:00 +0000 https://swsmith.cc/posts/intel-select-solution.html Intel Select Solution Cluster Setup AWS ParallelCluster is available as an Intel Select Solution for simulation and modeling. Configurations are verified to meet the standards set by the Intel HPC Platform Specification, use specific Intel instance types, and are configured to use the Elastic Fabric Adapter (EFA) networking interface. AWS ParallelCluster is the first cloud solution to meet the requirements for the Intel Select Solutions program. Supported instance types include c5n. https://swsmith.cc/posts/chrome-shortcuts.html Wed, 15 Feb 2017 00:00:00 +0000 https://swsmith.cc/posts/chrome-shortcuts.html I had this idea to make the browser more friendly to a keyboard native user like me. In the CLI I rely heavily on shortcuts aka “aliases” to navigate. So in 2017 I built a chrome extension that allows you to do this. The usage is simple, you navigate to a site you want to create a shortcut for, then add the page with a custom keyword: I put it up in the chrome webstore and forgot about it. https://swsmith.cc/about.html Mon, 01 Jan 0001 00:00:00 +0000 https://swsmith.cc/about.html I am a Solution Architect at AWS focused on AI/ML, specifically distributed training. My background is in High-Performance and Scientific Computing. I joined Amazon in 2017 and I helped build and launch AWS ParallelCluster, an open-source product to build High-Performance computing clusters. I then built out ParallelCluster Manager (now called ParallelCluster UI), a web-console for ParallelCluster. I’m a Pilot and a Flight Instructor (CFI, CFII). I own two airplanes and have flown to all 50 US States to raise money for Angel Flight West. https://swsmith.cc/bucketlist.html Mon, 01 Jan 0001 00:00:00 +0000 https://swsmith.cc/bucketlist.html What’s on your bucket list? ✅ Run the Ultra-trail Mt. Blanc (UTMB) proof ✅ Climb Mt Kilimanjaro ✅ Climb Mt Blanc proof ❌ Climb the Seven Highest Summits on each continent (2/7) ❌ Run a sub-3 hour marathon ❌ Ski tour the Haute route ✅ Learn to paraglide 🪂 ✅ Learn to fly ✈️ proof ❌ Climb the 5 volcanoes in WA (Rainier, Baker, Adams, St. Helens, Glacier Peak) (4/5) 🗻 ✅ Visit all 50 states ❌ Visit all 7 continents (4/7) ✅ Run 100 mile race proof ✅ Complete an Ironman proof ✅ Fly to Alaska proof ❌ Fly to Mexico Inspired by twogoglobal. https://swsmith.cc/slides.html Mon, 01 Jan 0001 00:00:00 +0000 https://swsmith.cc/slides.html Slides from various talks: Train and tune foundation models at scale with Amazon SageMaker, AWS NYC Summit - July 21th, 2024 Kanwaljit Khurmi and myself gave a talk at AWS NYC Summit in 2024 titled “Train and tune foundation models at scale with Amazon SageMaker” announcing a new product AWS Hyperpod. Design, engineering & simulation on AWS using HPC & virtual desktops, AWS re:Invent - November 29th, 2023 https://swsmith.cc/websites.html Mon, 01 Jan 0001 00:00:00 +0000 https://swsmith.cc/websites.html Websites I’ve built over the years: cluster-creator.swsmith.cc Cluster Creator makes the process of configuring AWS ParallelCluster easy. See Cluster Creator abexalate.com I built a website for my girlfriend Alex ❤️ blog.seanssmith.net Blog that I built completely from HTML + CSS. Chrome Shortcuts Chrome extension that grew to > 1K Monthly Active Users, even inspired a copy cat