MIT StarCluster

StarCluster Installation

Acquiring and configuring MIT StarCluster are both quite simple tasks. StarCluster is distributed as a Python package, called StarCluster. Thus, the usual Python package management tools, such as easy_install or pip, may be used to install it. For example:

easy_install StarCluster

You may also find that your particular Linux distribution has a package for StarCluster. For example, on Gentoo Linux, the package is named dev-python/starcluster. Using your Linux distribution’s package management system is usually preferable, unless you are installing an experimental version of StarCluster to a non-standard location. Your distribution’s package management system will help ensure that all necessary dependencies are properly built and working and make it easier to perform automatic upgrades in the future.

Please see the StarCluster intallation web page for more details: http://star.mit.edu/cluster/docs/latest/installation.html

StarCluster Configuration

Generating the Configuration File

When you run StarCluster for the first time, it will create a configuration subdirectory for you in your home directory, if such subdirectory does not already exist. This subdirectory is named .starcluster by default. To tell StarCluster to create this subdirectory, you can use something like the starcluster ls subcommand:

$ starcluster ls
StarCluster - (http://web.mit.edu/starcluster) (v. 0.93.3)
Software Tools for Academics and Researchers (STAR)
Please submit bug reports to starcluster@mit.edu

!!! ERROR - config file /home/em/.starcluster/config does not exist

Options:
--------
[1] Show the StarCluster config template
[2] Write config template to /home/em/.starcluster/config
[q] Quit

If everything was successful, then you should see that the ``.starcluster``
subdirectory contains a file named ``config``. This file was generated from a
template shipped with StarCluster.

The configuration file will need some tweaking before you can create a VM
instance suitable for use with Galaxy.

Please enter your selection: 2

>>> Config template written to /home/em/.starcluster/config
>>> Please customize the config template

You can verify the existence of your configuration subdirectory and its contents by listing them:

$ ls -l ~/.starcluster
total 12
-rw-r--r-- 1 em em 12106 Oct 16 10:19 config
drwxr-xr-x 2 em em    22 Oct 16 10:14 logs
drwxr-xr-x 2 em em     6 Oct 16 10:14 plugins

Note the presence of a file named config. This file was generated from a template that comes with the StarCluster software. By default, this file will not suffice for providing VM instances which are suitable for running Galaxy. We will need to edit it.

Editing the Global Configuration

The configuration file is monolithic; it contains your EC2 credentials, your desired cluster configurations, your desired firewall settings, your desired persistent data volumes, etc.... StarCluster provides an includes mechanism, which allows you to break out portions of this file into separate files. This is highly recommended, at least in the case of your EC2 credentials, to help ensure that you do not accidentally share them with someone else if you helpfully give a copy of your configuration to a colleague or friend. You may also use the inclusion mechanism simply to break the file up into more manageable pieces, if you desire.

At the top of the configuration file, there is a global section. Here, we are interested in changing two variables, DEFAULT_TEMPLATE and INCLUDE.

[global]
DEFAULT_TEMPLATE=solo
INCLUDE=~/...PRIVATE/starcluster-auth

We will create a new template named solo in a later section. (You can call it whatever you want, but solo will be used for the purpose of these instructions.) We will also create a separate directory and a separate file therein which will hold your EC2 credentials. To perform the first part of this latter task now, use the mkdir command with a secure access mode setting:

$ mkdir -m 700 ~/...PRIVATE

Securing Your EC2 Credentials

Back in the configuration file, you might notice that the next two sections concern your credentials and other information about your SSH keys. This is the very information which we want to exclude from the main configuration and place into a separate secure file. So, the following procedure is recommended:

  • Copy your configuration file over to your private, secure directory and name it starcluster-auth. Secure the file as well.

    $ install -m 600 ~/.starcluster/config ~/...PRIVATE/starcluster-auth
    

    Or, if the install command is not available to you:

    $ cp ~/.starcluster/config ~/...PRIVATE/starcluster-auth
    $ chmod 600 ~/...PRIVATE/starcluster-auth
    
  • Edit ~/...PRIVATE/starcluster-auth, removing all sections except the aws info section and any key sections.

  • Edit ~/...PRIVATE/starcluster-auth, filling out the aws info section and any key sections as appropriate. You will likely need to consult your Amazon EC2 console web page to gather some of the needed information. (This is a one-time activity. You should not need to use the web page after this.)

  • Edit ~/.starcluster/config, removing the aws info section and any key sections.

Defining a New Cluster

Next, we want to create a new cluster section in ~/.starcluster/config. Place the following in the file (preferably in the same area as the other cluster sections):

[cluster solo]
KEYNAME             = amazon-ged-ctb
CLUSTER_SIZE        = 1
CLUSTER_USER        = sgeadmin
CLUSTER_SHELL       = bash
NODE_IMAGE_ID       = ami-999d49f0
NODE_INSTANCE_TYPE  = m1.large
AVAILABILITY_ZONE   = us-east-1a
PERMISSIONS         = ssh, galaxy-http-main, galaxy-http-shed
# VOLUMES             = galaxy-devel

Warning

Please adjust the KEYNAME variable to match the name of a key section which you defined in your ~/...PRIVATE/starcluster-auth file.

Warning

If you are using a different EC2 availability zone than us-east-1a, then please alter the AVAILABILITY_ZONE variable accordingly.

Note

The StarCluster project may release new AMIs from time to time. Please consult the output of the starcluster listpublic subcommand for the current list.

You can adjust the NODE_INSTANCE_TYPE to fit your purposes and possibly save some money. A list of node instance types can be found in the cluster smallcluster section of the configuration file.

Warning

You will notice that the cluster smallcluster section is not commented out. Please either comment out this section or adjust the KEYNAME variable to the same value which you used in the cluster solo section.

The definition of the solo cluster has references to some firewall permissions that are not part of the configuration file by default. Likewise, there is a reference to a volume which needs to be defined. We will handle the definition of these things next.

Setting Firewall Permissions

If you scroll through the configuration file, you will eventually come across some permission sections. You will notice that there are a couple of permission http sections, a permission https section, and a permission ssh section. These are all commented out by default.

Remove the comments from in front of the permission ssh section. (You may leave the CIDR_IP assignment commented out, if you wish. You might want to do this if you intend to perform your development from a wide variety of locations with disparate IPs. If you intend to only develop from machines on the premises of a particular organization, such as a university, you can tighten down security by removing the comment from in front of the CIDR_IP assigment. CIDR stands for Classless Inter-Domain Routing. The short story is that every network on the Internet has a network number. The network number is calculated from an IP address and a netmask (network number mask). You can get the necessary information from your organization’s network administrators, if you don’t already know it and are uncomfortable crawling through Internet network registries (whois databases). You may also make an educated, overly-narrow guesstimate of your network number by taking your computer’s IP address (try hostname -i or /sbin/ifconfig), replacing the last of the four octets with 0, and placing a /24 after it. Or, you can just use your computer’s current IP address and place a /32 after it.)

For the purposes of running Galaxy, you do not need the standard web server ports (80 and 443). By default, Galaxy is configured to use ports 8080 and 9009. Port 8080 is, however, well-known as it gets used for a variety of auxiliary web services, particularly administration tools of various sorts. This makes it a target for intrusion attempts. If you plan on leaving your Galaxy server running for any length of time on EC2, then you will either want to configure Galaxy to use a different port number or you will want to set a CIDR_IP variable for it. (You can do both, of course.) These instructions will use ports 9007 and 9009. (If you find this choice of numbers to be odd, no one will dispute that.) Create the following two sections in your configuration file, using whichever port numbers you have decided upon (the port numbers should be greater than 1023):

# non-standard - 8080 is standard but intrusion attempts probe there
[permission galaxy-http-main]
PROTOCOL = tcp
FROM_PORT = 9007
TO_PORT = 9007
# CIDR_IP = <your_ip>/32

[permission galaxy-http-shed]
PROTOCOL = tcp
FROM_PORT = 9009
TO_PORT = 9009
# CIDR_IP = <your_ip>/32

Defining a Galaxy Volume

We need a place to persistently store Galaxy. Amazon provides persistent storage volumes known as EBS Volumes. The EBS (Elastic Block Store) service is another pay-as-you-go service like EC2. Since the size of the volume factors in to how much you pay, you want to optimize between price and having enough capacity for your needs. A reasonable starting configuration, which should accommodate software you might wish to install plus some sizable test data, might be 20 gigabytes.

Before an EBS volume can be referenced, it must exist. To create it, we will use:

$ starcluster cv -s 20 us-east-1a
StarCluster - (http://web.mit.edu/starcluster) (v. 0.93.3)
Software Tools for Academics and Researchers (STAR)
Please submit bug reports to starcluster@mit.edu

>>> No keypair specified, picking one from config...
>>> Using keypair: amazon-ged-ctb
>>> Creating security group @sc-volumecreator...
>>> No instance in group @sc-volumecreator for zone us-east-1a, launching one now.
Reservation:r-1156d977
>>> Waiting for volume host to come up... (updating every 30s)
>>> Waiting for all nodes to be in a 'running' state...
1/1 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%
>>> Waiting for SSH to come up on all nodes...
1/1 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 100%
>>> Waiting for cluster to come up took 1.551 mins
>>> Checking for required remote commands...
>>> Creating 20GB volume in zone us-east-1a
>>> New volume id: vol-a0cf0add
>>> Waiting for new volume to become 'available'...
>>> Attaching volume vol-a0cf0add to instance i-58fbf225...
>>> Formatting volume...
>>> Detaching volume vol-a0cf0add from volhost-us-east-1a
>>> Terminating node: volhost-us-east-1a (i-58fbf225)
>>> Waiting for cluster to terminate...
>>> Removing @sc-volumecreator security group

Note that if anything in the above process fails, then it means that you have something wrong with your configuration file and need to go back and carefully review the previous steps. If nothing failed, then congratulations, you just successfully created an EC2 instance and EBS volume and terminated the EC2 instance without using Amazon’s EC2 console.

Warning

The Amazon zone ‘us-east-1a’ is sometimes “full” and blocked from starting new machines. If you get an error that says something to this effect, then modify AVAILABILITY_ZONE in your .starcluster/config file to be something else, like ‘us-east-1d’. You will also need to modify the starcluster cv command to use ‘us-east-1d’.

Now, we need to edit the configuration file again. First, stop by the cluster solo section which you created earlier. Remove the comment from in front of the VOLUMES variable. After that, find the area where the volume sections reside and create a volume galaxy-devel section:

[volume galaxy-devel]
VOLUME_ID       = vol-a0cf0add
MOUNT_PATH      = /opt/sw/Galaxy

Warning

You need to set the VOLUME_ID variable to match the volume ID given in the output from your starcluster cv command. (In the above example, the volume ID is vol-a0cf0add.)

You can adjust the MOUNT_PATH variable to whatever you want. If the path to the mount point does not yet exist in the instance’s file system, then it will be automatically created by StarCluster before attempting to mount the volume there.

Miscellany

Your configuration of StarCluster should now be complete. To learn more about configuring this software, it is recommended that you read the helpful comments in the configuration file and that you consult http://star.mit.edu/cluster/docs/latest/manual/configuration.html

Using StarCluster

Getting Help

The starcluster help command is your friend. Note that you can get detailed help on subcommands by supplying the subcommand name as an additional argument. For example:

$ starcluster help cv
StarCluster - (http://web.mit.edu/starcluster) (v. 0.93.3)
Software Tools for Academics and Researchers (STAR)
Please submit bug reports to starcluster@mit.edu

Usage: createvolume [options] <volume_size> <volume_zone>

    Create a new EBS volume for use with StarCluster

The StarCluster manual is a useful resource, especially since it contains details that the built-in help does not provide. (E.g., the manual tells you what units the <volume_size> parameter uses for the createvolume subcommand.) The manual can be found on the web at http://star.mit.edu/cluster/docs/latest/contents.html

Short Example

To create an EC2 instance with the above configuration, the following command can be used, for example:

$ starcluster start -c solo galaxy
StarCluster - (http://web.mit.edu/starcluster) (v. 0.93.3)
Software Tools for Academics and Researchers (STAR)
Please submit bug reports to starcluster@mit.edu

>>> Validating cluster template settings...
>>> Cluster template settings are valid
>>> Starting cluster...
>>> Launching a 1-node cluster...
>>> Creating security group @sc-galaxy...
>>> Opening tcp port range 9009-9009 for CIDR 0.0.0.0/0
>>> Opening tcp port range 9007-9007 for CIDR 0.0.0.0/0
Reservation:r-abd1ded2
>>> Waiting for cluster to come up... (updating every 30s)

...

>>> Configuring cluster took 0.925 mins
>>> Starting cluster took 2.065 mins

To log into the newly created EC2 instance once it has finished starting, the following command can be used, for example:

$ starcluster sshmaster galaxy
StarCluster - (http://web.mit.edu/starcluster) (v. 0.93.3)
Software Tools for Academics and Researchers (STAR)
Please submit bug reports to starcluster@mit.edu

The authenticity of host 'ec2-54-234-37-131.compute-1.amazonaws.com
(54.234.37.131)' can't be established.
ECDSA key fingerprint is 4e:2c:7e:7c:08:1e:ed:df:7f:5d:4a:91:8b:ab:22:22.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added
'ec2-54-234-37-131.compute-1.amazonaws.com,54.234.37.131' (ECDSA) to the
list of known hosts.

...