WRF on the Cloud - Revision history

Zac at 17:43, 26 March 2021

2021-03-26T17:43:01Z

Show changes

Zac at 17:22, 26 March 2021

2021-03-26T17:22:48Z

Zac at 17:17, 26 March 2021

2021-03-26T17:17:59Z

Zac: /* Adding ssh users to a pcluster instance */

2020-12-30T16:01:20Z

‎Adding ssh users to a pcluster instance

Zac: /* Adding ssh users to a pcluster instance */

2020-12-30T16:00:53Z

‎Adding ssh users to a pcluster instance

Zac: /* Misc Notes */

2020-12-30T15:50:29Z

‎Misc Notes

Zac: /* Increase size of in-use volume */

2019-12-17T17:49:00Z

‎Increase size of in-use volume

Zac: /* AWS Parallel Cluster */

2019-10-19T03:54:08Z

‎AWS Parallel Cluster

Zac: /* AWS Parallel Cluster */

2019-07-01T18:40:46Z

‎AWS Parallel Cluster

Zac: /* AWS Parallel Cluster */

2019-06-12T16:24:51Z

‎AWS Parallel Cluster

← Older revision		Revision as of 17:22, 26 March 2021
Line 25:		Line 25:
	* Production done with EC2 spot c4* instances (see spot pcluster config below) and sc1 volumes		* Production done with EC2 spot c4* instances (see spot pcluster config below) and sc1 volumes

−	~~== AWS Parallel Cluster ==~~
−	* [https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html Install AWS-CLI]
−	* [https://aws.amazon.com/blogs/opensource/aws-parallelcluster/ Install Pcluster]
−	* Don't set the spot price. When you do not set a spot price, AWS will give you the [https://aws.amazon.com/blogs/compute/new-amazon-ec2-spot-pricing/ spot market price capped at the on-demand price]. Setting the spot price makes the instance more prone to being reclaimed and having your job terminated. As there is currently no functionality automatically enabled on EC2 instances for checkpoint/restart, losing an instance is a show-stopper for WRF production runs. For WRF MPI applications it's not worth playing the spot market if the tradeoff is instance reliability.
−	* Use the base alinux AMI for your version of pcluster; e.g., for v2.4.0 : https://github.com/aws/aws-parallelcluster/blob/v2.4.0/amis.txt
−	~~Configure the cluster with a config file:~~
−
−	~~=== spot pcluster config ===~~
−
−	~~[aws]~~
−	~~aws_region_name = us-east-2~~
−
−	~~[cluster ladcospot]~~
−	~~vpc_settings = public~~
−	~~ebs_settings = ladcosc1~~
−	~~scheduler = sge~~
−	~~master_instance_type = c4.large~~
−	~~compute_instance_type = c4.4xlarge~~
−	~~placement = computer~~
−	~~placement_group = DYNAMIC~~
−	~~master_root_volume_size = 40~~
−	~~cluster_type = spot~~
−	~~#spot_price = 0.2~~
−	~~base_os = alinux~~
−	key_name = *****
−	~~# Base AMI for pcluster v2.1.0~~
−	~~custom_ami = ami-0381cb7486cdc973f~~
−
−	~~# Create a cold storage I/O directory~~
−	~~[ebs ladcosc1]~~
−	~~shared_dir = data~~
−	~~volume_type = sc1~~
−	~~volume_size = 10000~~
−	~~volume_iops = 1500~~
−	~~encrypted = false~~
−
−	~~[vpc public]~~
−	master_subnet_id = subnet-******
−	vpc_id = vpc-******
−
−	~~[global]~~
−	~~update_check = true~~
−	~~sanity_check = true~~
−	~~cluster_template = ladcospot~~
−
−	~~[aliases]~~
−	~~ssh = ssh -Y {CFN_USER}@{MASTER_IP} {ARGS}~~
−
−	~~=== on demand pcluster config ===~~
−
−	~~[aws]~~
−	~~aws_region_name = us-east-2~~
−
−	~~[cluster ladcowrf]~~
−	~~vpc_settings = public~~
−	~~ebs_settings = ladcowrf~~
−	~~scheduler = sge~~
−	~~master_instance_type = m4.large~~
−	~~compute_instance_type = m5a.4xlarge~~
−	~~placement = cluster~~
−	~~placement_group = DYNAMIC~~
−	~~master_root_volume_size = 40~~
−	~~cluster_type = ondemand~~
−	~~base_os = alinux~~
−	key_name = *****
−	~~min_vcpus = 0~~
−	~~max_vcpus = 64~~
−	~~desired_vcpus = 0~~
−	~~# Base AMI for pcluster v2.1.0~~
−	~~custom_ami = ami-0381cb7486cdc973f~~
−
−	~~[ebs ladcowrf]~~
−	~~shared_dir = data~~
−	~~volume_type = gp2~~
−	~~volume_size = 10000~~
−	~~volume_iops = 1500~~
−	~~encrypted = false~~
−
−	~~[vpc public]~~
−	master_subnet_id = subnet-******
−	vpc_id = vpc-******
−
−	~~[global]~~
−	~~update_check = true~~
−	~~sanity_check = true~~
−	~~cluster_template = ladcowrf~~
−
−	~~[aliases]~~
−	~~ssh = ssh -Y {CFN_USER}@{MASTER_IP} {ARGS}~~
−
−	~~=== Cluster Access ===~~
−
−	~~Start the cluster~~
−	~~pcluster create -c config.spot ladcospot~~
−
−	~~Log in to the cluster~~
−	~~pcluster ssh ladcospot -i {name of your AWS Key}~~
−
−	~~=== Fault Tolerance ===~~
−
−	~~Script to monitor pcluster system logs for termination notice, and restart WRF.~~
−
−	~~#!/bin/bash~~
−	~~CASE=LADCO_2016_WRFv39_YNT_NAM~~
−	~~JSTART=2016095~~
−	~~wrk_dir=/data/apps/WRFV3.9.1/sims/LADCO_2016_WRFv39_YNT_NAM~~
−	~~while true~~
−	do
−	~~if [ -z $(curl -s http://169.254.169.254/latest/meta-data/spot/termination-time \| grep -q .T.Z ) ]; then~~
−	~~echo "terminated"~~
−	~~break~~
−	~~else~~
−	~~echo "Still running fine"~~
−	~~sleep 3~~
−	fi
−	~~done~~
−	~~echo "Restarting WRF job"~~
−	~~#qsub -N WRF_rest $wrk_dir/wrapper_restart_wrf.csh $JSTART 6~~
−	~

	== AWS Configuration ==		== AWS Configuration ==

← Older revision		Revision as of 17:17, 26 March 2021
Line 1:		Line 1:

−	~~= Objectives =~~	+	[[Ramboll Modeling on the Cloud Contract Results]]
−	LADCO is seeking to understand the best practices for submitting and managing multiprocessor computing jobs on a cloud computing platform. In particular, LADCO would like to develop a WRF production environment that utilizes cloud-based computing. The goal of this project is to prototype a WRF production environment on a public, on~~-demand high performance computing service in the cloud to create a WRF platform-as-a-service (PaaS) solution. The WRF PaaS must meet~~ the ~~following objectives:~~

−	* Configurable computing and storage to scale, as needed, to meet that needs of different WRF applications
−	* Configurable WRF options to enable changing grids, simulation periods, physics options, and input data
−	* Flexible cloud deployment from a command line interface to initiate computing clusters and spawn WRF jobs in the cloud
−
−	~~= Call Notes =~~
−	~~== November 28, 2018 ==~~
−	~~=== WRF Benchmarking ===~~
−	* Emulating WRF 2016 12/4/1.3 grids
−	* Purpose for estimating costing for CPUs, RAM and Storage
−	* CPU: 8 Cores: 5.5 day run = 4 days; 24 Cores: 3 days
−	* RAM: ~22 Gb RAM/run (2.5 Gb/core)
−	* Storage
−	** test netCDF4 and netCDF with no compression
−	** with compression saves a lot of space (1/3 of the output) relative to uncompressed NCF (~70% compression)
−	** need to link in the HDF and NC4 libraries with compression to downstream programs
−	** estimate about 5.8 Tb for the year, goes to 16.9 Tb without compression
−
−	~~=== Conceptual Approach to WRF on the Cloud ===~~
−	* Cluster management would launch a head node and compute nodes
−	* 77 5.5 day chunks, 20 computers for 16 days (or 80 computers for 4 days)
−	* Head node running constantly
−	* Compute nodes running over the length of project
−	* Memory optimized machines performed better than compute optimized for CAMx
−
−	~~=== Cost Analysis ===~~
−	* [https://www.ladco.org/wp-content/uploads/Projects/WRF-Cloud/WRF_cloud_computing_costs.pdf Analysis Spreadsheet]
−
−	~~=== Storage Analysis ===~~
−	* AWS
−	** Don't want to use local because it will need to be moved/migrated
−	** Put the data on a storage appliance (S3) while running, and then push off to longer term storage (Glacier)
−	** Glacier is archived and need to submit access through the console, response times listed as 1-5 minutes
−	* Azure
−	** Fast and slower lake storage for offline
−	** Managed disks for online
−
−	~~=== Data Transfer Analyis ===~~
−	* estimate based on 5.8 Gb
−	* AWS
−	** Internet transfer will cost ~ $928 for 5.5 Gb
−	** Snowball 10 days to get data off a disk, costs $200 for entire WRF run (smallest was 50 Tb)
−	* Azure
−	** Online transfer
−	** Databox option (like snowball)
−
−	~~=== Cluster Management Tools (interface analysis) ===~~
−	* 3-4 seemed to work best across several cloud solutions
−	* Alsys Flight (works on AWS and Azure), used to bring up 40 nodes; set up a Tor queuing system; trouble with using an AMI, need to pay for an AMI with this solution; can use Docker if we want to use containers, but Ramboll not positioned to use containers for this project
−	* CFN: slower development, but now has an AWS parallel cluster (CFN reincarnated), improved tools and built in the Python package index (can be installed with PIP); let's you spin everything up from the command line and could be scripted
−	* Haven't yet explored AWS Parallel Cluster/CFN in detail; similar to experience with Star Cluster; seems to be the best solution because you can use your own custom AMI; instance types are independent of the cluster management tools
−
−	~~=== Next Steps ===~~
−	* LADCO to create a WRF AMI on AWS: WRF 3.9.1, netCDF4 with compression, MPICH2, PGI compiler, AMET
−	* LADCO to create a login for Ramboll in our AWS organization
−	* Ramboll to explore AWS Parallel cluster and then prototype with LADCO WRF AMI
−	* Next call 12/5 @ 3 Central
−
−	~~= Ramboll Recommendations =~~
−	~~== WRF ==~~
−	* Use netCDF4 with compression
−	* Use 8 cores per 5.5-day segment and submit all segments for annual run to cluster at once
−	~~== Cloud Service ==~~
−	* Costs are equivalent between Azure and AWS so use AWS because of familiarity
−	* Use one memory optimized instance (EC2-r5.2xlarge, 8 cores, 64 GB RAM) for each segment
−	* Use Standard S3 storage for the lifetime of the project and migrate to Infrequent S3 or Glacier for longterm storage
−	* Use Snowball to transfer completed project to local site
−
−	~~== HPC Platforms ==~~
−	* Use AWS ParallelCluster (formerly CfnCluster)
−	** Provides CLI-interface, allowing for linux-script automation
−	** Allows for custom AMIs
−	** Provides a variety of schedulers: sge, torque, slurm, or awsbatch
−	** Is actively being developed and enhanced
−	** Additional investigation/test of WRF/CAMx test cases needed to verify tool integrity and performance
−	* Other HPC have demonstrated issues
−	** StarCluster: Problematic auto-scaling; outdated and inactive
−	** AlcesFlight: Fee-based ability to use custom AMIs, problems with auto-scaling for large instance counts

	= WRF on AWS: LADCO User's Guide =		= WRF on AWS: LADCO User's Guide =

@@ Line 243: / Line 243: @@
 The post-install command in the pcluster configuration file should invoke the settings in the post_install_users.sh script described above.  Alternatively, if you already have an instance running you can run the script as sudo to invoke the settings:
-    sudo ./post_install_users.sh
+    > sudo ./post_install_users.sh
 After running the script you need to use the add-key.sh and add-user.sh script described in the above link.
    #Usage (using zac as an example)
-   sudo ./add-user.sh zac {####., e.g, 2001}
+   > sudo ./add-user.sh zac {####., e.g, 2001}
-   sudo ./add-key.sh zac ladco.key
+   > sudo ./add-key.sh zac ladco.key
    #Add users to /etc/passwd file
-   sudo getent passwd zac
+   > sudo getent passwd zac
-   sudo vi /etc/passwd
+   > sudo vi /etc/passwd
    #Change ladco to primary group
-   sudo usermod zac -g ladco
+   > sudo usermod zac -g ladco
 ===Using AWS S3 for offline storage===

← Older revision		Revision as of 16:00, 30 December 2020
Line 240:		Line 240:

	[https://aws.amazon.com/blogs/opensource/managing-aws-parallelcluster-ssh-users-with-openldap/ AWS documentation for adding users with OpenLDAP]		[https://aws.amazon.com/blogs/opensource/managing-aws-parallelcluster-ssh-users-with-openldap/ AWS documentation for adding users with OpenLDAP]
		+
		+	The post-install command in the pcluster configuration file should invoke the settings in the post_install_users.sh script described above. Alternatively, if you already have an instance running you can run the script as sudo to invoke the settings:
		+
		+	sudo ./post_install_users.sh
		+
		+	After running the script you need to use the add-key.sh and add-user.sh script described in the above link.
		+
		+	#Usage (using zac as an example)
		+	sudo ./add-user.sh zac {####., e.g, 2001}
		+	sudo ./add-key.sh zac ladco.key
		+	#Add users to /etc/passwd file
		+	sudo getent passwd zac
		+	sudo vi /etc/passwd
		+	#Change ladco to primary group
		+	sudo usermod zac -g ladco

	===Using AWS S3 for offline storage===		===Using AWS S3 for offline storage===

← Older revision		Revision as of 15:50, 30 December 2020
Line 236:		Line 236:

	== Misc Notes ==		== Misc Notes ==
		+
		+	===Adding ssh users to a pcluster instance===
		+
		+	[https://aws.amazon.com/blogs/opensource/managing-aws-parallelcluster-ssh-users-with-openldap/ AWS documentation for adding users with OpenLDAP]

	===Using AWS S3 for offline storage===		===Using AWS S3 for offline storage===

← Older revision		Revision as of 17:49, 17 December 2019
Line 264:		Line 264:


−	===[https://~~support~~.~~amimoto-ami~~.com/~~english~~/~~self-hosting-accounts~~/~~increasing~~-~~your-ec2~~-volume-~~size~~ Increase size of in-use volume]===	+	===[https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/recognize-expanded-volume-linux.html Increase size of in-use volume]===

	===Add New Volume to Running Instance===		===Add New Volume to Running Instance===

@@ Line 200: / Line 200: @@
 Log in to the cluster
   pcluster ssh ladcospot -i {name of your AWS Key}
 == AWS Configuration ==

@@ Line 107: / Line 107: @@
 * [https://aws.amazon.com/blogs/opensource/aws-parallelcluster/ Install Pcluster]
 * Don't set the spot price.  When you do not set a spot price, AWS will give you the [https://aws.amazon.com/blogs/compute/new-amazon-ec2-spot-pricing/ spot market price capped at the on-demand price]. Setting the spot price makes the instance more prone to being reclaimed and having your job terminated.  As there is currently no functionality automatically enabled on EC2 instances for checkpoint/restart, losing an instance is a show-stopper for WRF production runs. For WRF MPI applications it's not worth playing the spot market if the tradeoff is instance reliability.
+* Use the base alinux AMI for your version of pcluster; e.g., for v2.4.0 : https://github.com/aws/aws-parallelcluster/blob/v2.4.0/amis.txt
 Configure the cluster with a config file:

@@ Line 106: / Line 106: @@
 * [https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html Install AWS-CLI]
 * [https://aws.amazon.com/blogs/opensource/aws-parallelcluster/ Install Pcluster]
+* Don't set the spot price.  When you do not set a spot price, AWS will give you the [https://aws.amazon.com/blogs/compute/new-amazon-ec2-spot-pricing/ spot market price capped at the on-demand price]. Setting the spot price makes the instance more prone to being reclaimed and having your job terminated.  As there is currently no functionality automatically enabled on EC2 instances for checkpoint/restart, losing an instance is a show-stopper for WRF production runs. For WRF MPI applications it's not worth playing the spot market if the tradeoff is instance reliability.
 Configure the cluster with a config file:
@@ Line 124: / Line 125: @@
   master_root_volume_size = 40
   cluster_type = spot
-  spot_price = 0.2
+  #spot_price = 0.2
   base_os = alinux
   key_name = *****