Package edu.isi.pegasus.planner.refiner
Class DeployWorkerPackage
- java.lang.Object
-
- edu.isi.pegasus.planner.refiner.Engine
-
- edu.isi.pegasus.planner.refiner.DeployWorkerPackage
-
public class DeployWorkerPackage extends Engine
The refiner that is responsible for adding - setup nodes that deploy a worker package on each deployment site at start of workflow execution - cleanup nodes that undeploy a worker package on each deployment site at end workflow execution- Version:
- $Revision: 538 $
- Author:
- Karan Vahi, Gaurang Mehta
-
-
Field Summary
Fields Modifier and Type Field Description static java.lang.String
BASE_BUILD_DIRECTORY_URL
The base directory URL for the builds.static java.lang.String
CLEANUP_PREFIX
Constant suffix for the names of the deployment nodes.static java.lang.String
COMPLETE_TRANSFORMATION_NAME
The complete TC name for pegasus worker package.static java.lang.String
COMPLETE_UNTAR_TRANSFORMATION_NAME
The complete TC name for untar.static java.lang.String
DEFAULT_REFINER
The default transfer refiner name.static java.lang.String
DEPLOY_WORKER_PREFIX
Constant suffix for the names of the deployment nodes.static java.lang.String
DERIVATION_NAME
The logical name of the transformation for the worker packagestatic java.lang.String
DERIVATION_NAMESPACE
The derivation namespace for the worker package.static java.lang.String
DERIVATION_VERSION
The version number for the derivations for worker package.protected ADag
mCurrentDag
It is a reference to the Concrete Dag so far.protected java.util.Map<java.lang.String,FileTransfer>
mFTMap
The FileTransfer map indexed by site id.protected java.lang.String
mJobPrefix
The job prefix that needs to be applied to the job file basenames.protected java.util.Map<java.lang.String,java.lang.Boolean>
mLocalTransfers
Map that indicates whether we need local setup transfer jobs for a site or not.private static java.util.Map<SysInfo.OS,java.lang.String>
mOSToNMIOSReleaseAndVersion
The map storing OS to corresponding NMI OS platforms.private static java.util.regex.Pattern
mPattern
Stores compiled patterns at first use, quasi-Singleton.private static java.lang.String
mRegexExpression
Store the regular expressions necessary to parse the basename from the worker package url to retrieve the version of pegasus.protected Implementation
mSetupTransferImplementation
The transfer implementation to be used for staging in the data as part of setup job.protected java.util.Map<java.lang.String,java.lang.String>
mSiteToPegasusHomeMap
Maps a site to the the directory where the pegasus worker package has been untarred during workflow execution.private static java.util.Set<java.lang.String>
mSupportedOSReleaseVersions
A set of supported OS release and versions that our build process builds for.protected boolean
mTransferWorkerPackage
Boolean indicating whether user wants the worker package to be transferred or not.protected java.lang.String
mUserSpecifiedSourceLocation
The user specified location from where to stage the worker packages.protected boolean
mUseUserSpecifiedSourceLocation
Boolean indicating whether to use the user specified location or notprotected boolean
mWorkerNodeExecution
Boolean indicating worker node execution.static java.lang.String
PACKAGE_NAME
The name of the package in which all the implementing classes are.static java.lang.String
PATH_VALUE
The path to be set for create dir jobs.static java.lang.String
PEGASUS_VERSION
The version of pegasus matching the planner.static java.lang.String[][]
PEGASUS_WORKER_EXECUTABLES
Array storing the names of the executables in the $PEGASUS_HOME/bin directory Associates the transformation name with the executable basenamesstatic java.lang.String
POSTSCRIPT_ARGUMENTS_FOR_ONLY_ROTATING_LOG_FILE
The arguments for pegasus-exitcode when you only want the log files to be rotated.static java.lang.String
REFINER_NAME
The name of the refiner for purposes of error loggingstatic java.lang.String
TRANSFORMATION_NAME
The logical name of the worker packagestatic java.lang.String
TRANSFORMATION_NAMESPACE
The transformation namespace for the worker packagestatic java.lang.String
TRANSFORMATION_VERSION
The version number for the worker package.static java.lang.String
UNTAR_DERIVATION_NAME
The logical name of the transformation for the untar job.static java.lang.String
UNTAR_DERIVATION_NAMESPACE
The derivation namespace for the untar job.static java.lang.String
UNTAR_DERIVATION_VERSION
The version number for the derivations for untar job.static java.lang.String
UNTAR_PREFIX
Constant suffix for the names of the deployment nodes.static java.lang.String
UNTAR_TRANSFORMATION_NAME
The logical name of the worker packagestatic java.lang.String
UNTAR_TRANSFORMATION_NAMESPACE
The transformation namespace for the worker packagestatic java.lang.String
UNTAR_TRANSFORMATION_VERSION
The version number for the worker package.-
Fields inherited from class edu.isi.pegasus.planner.refiner.Engine
mBag, mLogger, mLogMsg, mOutputPool, mPoolFile, mPOptions, mProps, mRLIUrl, mSiteStore, mTCFile, mTCHandle, mTCMode, REGISTRATION_UNIVERSE, TRANSFER_UNIVERSE
-
-
Constructor Summary
Constructors Constructor Description DeployWorkerPackage(PegasusBag bag)
A pratically nothing constructor !
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description ADag
addCleanupNodesForWorkerPackage(ADag dag)
Adds cleanup nodes in the workflow for sites specified.private TransformationCatalogEntry
addDefaultTCEntry(java.lang.String site, java.lang.String pegasusHome, SysInfo sysinfo, boolean useFullPath, java.lang.String name, java.lang.String executable)
Returns a default TC entry to be used in case entry is not found in the transformation catalog.protected TransformationCatalogEntry
addDefaultTCEntryForPegasusWebsite(java.lang.String site, java.lang.String name)
Returns a default TC entry for the pegasus site.ADag
addSetupNodes(ADag dag)
Adds a setup node per execution site in the workflow that will stage the worker node executables to the workdirectory on the sites the workflow has been scheduled to.private ADag
addSetupNodesWithoutUntarNodes(ADag dag, java.util.Set<java.lang.String> deploymentSites)
Adds only the stage worker nodes to the workflow.private ADag
addSetupNodesWithUntarNodes(ADag dag, java.util.Set<java.lang.String> deploymentSites)
Adds untar nodes to the workflow, in addition to the stage worker nodesprotected java.lang.String
constructDefaultURLToPegasusWorkerPackage(java.lang.String name, SysInfo sysinfo)
Constructs the default URL's for the pegasus worker package.private TransformationCatalogEntry
defaultUntarTCEntry(SiteCatalogEntry site)
Returns a default TC entry to be used in case entry is not found in the transformation catalog.protected java.lang.String
getBasename(java.lang.String url)
Returns the basename of the URL using substring.protected java.lang.String
getCleanupJobname(ADag dag, java.lang.String site)
It returns the name of the untar job, that is to be assigned.protected java.lang.String
getDeployJobName(ADag dag, java.lang.String site, boolean localTransfer)
It returns the name of the deployment job, that is to be assigned.protected java.util.Set
getDeploymentSites(ADag dag)
Retrieves the sites for which the deployment jobs need to be created.protected java.lang.String
getRootDirectoryNameForPegasus(java.lang.String url)
Does regex magic to figure out the version of pegasus from the url, and use it to construct the name of pegasus directory, when worker package is untarred.private java.lang.String
getStagingSite(java.lang.String site)
Returns the staging site for a particular execution site.protected java.lang.String
getUntarJobName(ADag dag, java.lang.String site)
It returns the name of the untar job, that is to be assigned.void
initialize(ADag scheduledDAG)
Initialize with the scheduled graph.static DeployWorkerPackage
loadDeployWorkerPackage(PegasusBag bag)
Loads the implementing class corresponding to the mode specified by the user at runtime.protected Job
makeUntarJob(java.lang.String site, java.lang.String jobName, java.lang.String wpBasename)
It creates a untar job , that untars the worker package that is staged by the setup transfer job.private static java.util.Map<SysInfo.OS,java.lang.String>
osToOSReleaseAndVersion()
Maps each to OS to a specific OS release for purposes of picking up the correct worker package for a site.boolean
runTransferOnLocalSite(Refiner refiner, java.lang.String site, java.lang.String destinationURL, int type)
Returns whether to run a transfer job on local site or not.private static java.util.Set<java.lang.String>
supportedOSReleaseAndVersions()
A set of OS release and version combinations for which our build processes build Pegasus binaries.-
Methods inherited from class edu.isi.pegasus.planner.refiner.Engine
addVector, appendArrayList, complainForHeadNodeURLPrefix, complainForHeadNodeURLPrefix, loadProperties, printVector, stringInList, stringInPegVector, stringInVector, vectorToString
-
-
-
-
Field Detail
-
DEPLOY_WORKER_PREFIX
public static final java.lang.String DEPLOY_WORKER_PREFIX
Constant suffix for the names of the deployment nodes.- See Also:
- Constant Field Values
-
UNTAR_PREFIX
public static final java.lang.String UNTAR_PREFIX
Constant suffix for the names of the deployment nodes.- See Also:
- Constant Field Values
-
CLEANUP_PREFIX
public static final java.lang.String CLEANUP_PREFIX
Constant suffix for the names of the deployment nodes.- See Also:
- Constant Field Values
-
POSTSCRIPT_ARGUMENTS_FOR_ONLY_ROTATING_LOG_FILE
public static final java.lang.String POSTSCRIPT_ARGUMENTS_FOR_ONLY_ROTATING_LOG_FILE
The arguments for pegasus-exitcode when you only want the log files to be rotated.- See Also:
- Constant Field Values
-
PEGASUS_WORKER_EXECUTABLES
public static final java.lang.String[][] PEGASUS_WORKER_EXECUTABLES
Array storing the names of the executables in the $PEGASUS_HOME/bin directory Associates the transformation name with the executable basenames
-
mRegexExpression
private static final java.lang.String mRegexExpression
Store the regular expressions necessary to parse the basename from the worker package url to retrieve the version of pegasus.- See Also:
- Constant Field Values
-
PATH_VALUE
public static final java.lang.String PATH_VALUE
The path to be set for create dir jobs.- See Also:
- Constant Field Values
-
DEFAULT_REFINER
public static final java.lang.String DEFAULT_REFINER
The default transfer refiner name.- See Also:
- Constant Field Values
-
TRANSFORMATION_NAMESPACE
public static final java.lang.String TRANSFORMATION_NAMESPACE
The transformation namespace for the worker package- See Also:
- Constant Field Values
-
TRANSFORMATION_NAME
public static final java.lang.String TRANSFORMATION_NAME
The logical name of the worker package- See Also:
- Constant Field Values
-
TRANSFORMATION_VERSION
public static final java.lang.String TRANSFORMATION_VERSION
The version number for the worker package.
-
UNTAR_TRANSFORMATION_NAMESPACE
public static final java.lang.String UNTAR_TRANSFORMATION_NAMESPACE
The transformation namespace for the worker package
-
UNTAR_TRANSFORMATION_NAME
public static final java.lang.String UNTAR_TRANSFORMATION_NAME
The logical name of the worker package- See Also:
- Constant Field Values
-
UNTAR_TRANSFORMATION_VERSION
public static final java.lang.String UNTAR_TRANSFORMATION_VERSION
The version number for the worker package.
-
COMPLETE_UNTAR_TRANSFORMATION_NAME
public static final java.lang.String COMPLETE_UNTAR_TRANSFORMATION_NAME
The complete TC name for untar.
-
COMPLETE_TRANSFORMATION_NAME
public static final java.lang.String COMPLETE_TRANSFORMATION_NAME
The complete TC name for pegasus worker package.
-
DERIVATION_NAMESPACE
public static final java.lang.String DERIVATION_NAMESPACE
The derivation namespace for the worker package.- See Also:
- Constant Field Values
-
DERIVATION_NAME
public static final java.lang.String DERIVATION_NAME
The logical name of the transformation for the worker package- See Also:
- Constant Field Values
-
DERIVATION_VERSION
public static final java.lang.String DERIVATION_VERSION
The version number for the derivations for worker package.- See Also:
- Constant Field Values
-
UNTAR_DERIVATION_NAMESPACE
public static final java.lang.String UNTAR_DERIVATION_NAMESPACE
The derivation namespace for the untar job.
-
UNTAR_DERIVATION_NAME
public static final java.lang.String UNTAR_DERIVATION_NAME
The logical name of the transformation for the untar job.- See Also:
- Constant Field Values
-
UNTAR_DERIVATION_VERSION
public static final java.lang.String UNTAR_DERIVATION_VERSION
The version number for the derivations for untar job.- See Also:
- Constant Field Values
-
PACKAGE_NAME
public static final java.lang.String PACKAGE_NAME
The name of the package in which all the implementing classes are.- See Also:
- Constant Field Values
-
BASE_BUILD_DIRECTORY_URL
public static final java.lang.String BASE_BUILD_DIRECTORY_URL
The base directory URL for the builds.- See Also:
- Constant Field Values
-
PEGASUS_VERSION
public static final java.lang.String PEGASUS_VERSION
The version of pegasus matching the planner.
-
REFINER_NAME
public static final java.lang.String REFINER_NAME
The name of the refiner for purposes of error logging- See Also:
- Constant Field Values
-
mPattern
private static java.util.regex.Pattern mPattern
Stores compiled patterns at first use, quasi-Singleton.
-
mOSToNMIOSReleaseAndVersion
private static java.util.Map<SysInfo.OS,java.lang.String> mOSToNMIOSReleaseAndVersion
The map storing OS to corresponding NMI OS platforms.
-
mSupportedOSReleaseVersions
private static java.util.Set<java.lang.String> mSupportedOSReleaseVersions
A set of supported OS release and versions that our build process builds for.
-
mCurrentDag
protected ADag mCurrentDag
It is a reference to the Concrete Dag so far.
-
mJobPrefix
protected java.lang.String mJobPrefix
The job prefix that needs to be applied to the job file basenames.
-
mSetupTransferImplementation
protected Implementation mSetupTransferImplementation
The transfer implementation to be used for staging in the data as part of setup job.
-
mFTMap
protected java.util.Map<java.lang.String,FileTransfer> mFTMap
The FileTransfer map indexed by site id.
-
mLocalTransfers
protected java.util.Map<java.lang.String,java.lang.Boolean> mLocalTransfers
Map that indicates whether we need local setup transfer jobs for a site or not.
-
mSiteToPegasusHomeMap
protected java.util.Map<java.lang.String,java.lang.String> mSiteToPegasusHomeMap
Maps a site to the the directory where the pegasus worker package has been untarred during workflow execution.
-
mUserSpecifiedSourceLocation
protected java.lang.String mUserSpecifiedSourceLocation
The user specified location from where to stage the worker packages.
-
mUseUserSpecifiedSourceLocation
protected boolean mUseUserSpecifiedSourceLocation
Boolean indicating whether to use the user specified location or not
-
mTransferWorkerPackage
protected boolean mTransferWorkerPackage
Boolean indicating whether user wants the worker package to be transferred or not.
-
mWorkerNodeExecution
protected boolean mWorkerNodeExecution
Boolean indicating worker node execution.
-
-
Constructor Detail
-
DeployWorkerPackage
public DeployWorkerPackage(PegasusBag bag)
A pratically nothing constructor !- Parameters:
bag
- bag of initialization objects
-
-
Method Detail
-
osToOSReleaseAndVersion
private static java.util.Map<SysInfo.OS,java.lang.String> osToOSReleaseAndVersion()
Maps each to OS to a specific OS release for purposes of picking up the correct worker package for a site. The mapping is to be kept consistent with the NMI builds for the releases.- Returns:
- map
-
supportedOSReleaseAndVersions
private static java.util.Set<java.lang.String> supportedOSReleaseAndVersions()
A set of OS release and version combinations for which our build processes build Pegasus binaries.- Returns:
-
loadDeployWorkerPackage
public static DeployWorkerPackage loadDeployWorkerPackage(PegasusBag bag) throws FactoryException
Loads the implementing class corresponding to the mode specified by the user at runtime.- Parameters:
bag
- bag of initialization objects- Returns:
- instance of a DeployWorkerPackage implementation
- Throws:
FactoryException
- that nests any error that might occur during the instantiation of the implementation.
-
initialize
public void initialize(ADag scheduledDAG)
Initialize with the scheduled graph. Results in the appropriate population of the transformation catalog with pegasus-worker executables.- Parameters:
scheduledDAG
- the scheduled workflow.
-
runTransferOnLocalSite
public boolean runTransferOnLocalSite(Refiner refiner, java.lang.String site, java.lang.String destinationURL, int type)
Returns whether to run a transfer job on local site or not.- Parameters:
site
- the site handle associated with the destination URL.destURL
- the destination URLtype
- the type of transfer job for which the URL is being constructed.- Returns:
- true indicating if the associated transfer job should run on local site or not.
-
getRootDirectoryNameForPegasus
protected java.lang.String getRootDirectoryNameForPegasus(java.lang.String url)
Does regex magic to figure out the version of pegasus from the url, and use it to construct the name of pegasus directory, when worker package is untarred.- Parameters:
url
- the url.- Returns:
- basename for pegasus directory
-
addSetupNodes
public ADag addSetupNodes(ADag dag)
Adds a setup node per execution site in the workflow that will stage the worker node executables to the workdirectory on the sites the workflow has been scheduled to.- Parameters:
dag
- the scheduled workflow.- Returns:
- the workflow with setup jobs added
-
addSetupNodesWithUntarNodes
private ADag addSetupNodesWithUntarNodes(ADag dag, java.util.Set<java.lang.String> deploymentSites)
Adds untar nodes to the workflow, in addition to the stage worker nodes- Parameters:
dag
- the dagdeploymentSites
- the sites for which the worker package has to be deployed- Returns:
- the workflow in the graph representation with the nodes added.
-
addSetupNodesWithoutUntarNodes
private ADag addSetupNodesWithoutUntarNodes(ADag dag, java.util.Set<java.lang.String> deploymentSites)
Adds only the stage worker nodes to the workflow. This is used when Pegasus Lite is used to launch the jobs on the execution sites.- Parameters:
dag
- the dagdeploymentSites
- the sites for which the worker package has to be deployed- Returns:
- the workflow in the graph representation with the nodes added.
-
addCleanupNodesForWorkerPackage
public ADag addCleanupNodesForWorkerPackage(ADag dag)
Adds cleanup nodes in the workflow for sites specified.- Parameters:
dag
- the workflow- Returns:
- workflow with cleanup jobs added
-
getDeploymentSites
protected java.util.Set getDeploymentSites(ADag dag)
Retrieves the sites for which the deployment jobs need to be created.- Parameters:
dag
- the dag on which the jobs need to execute.- Returns:
- a Set containing a list of siteID's of the sites where the dag has to be run.
-
getDeployJobName
protected java.lang.String getDeployJobName(ADag dag, java.lang.String site, boolean localTransfer)
It returns the name of the deployment job, that is to be assigned. The name takes into account the workflow name while constructing it, as that is thing that can guarentee uniqueness of name in case of deferred planning.- Parameters:
dag
- the workflow so far.site
- the execution pool for which the create directory job is responsible.localTransfer
- whether the transfer needs to run locally or not.- Returns:
- String corresponding to the name of the job.
-
getUntarJobName
protected java.lang.String getUntarJobName(ADag dag, java.lang.String site)
It returns the name of the untar job, that is to be assigned. The name takes into account the workflow name while constructing it, as that is thing that can guarentee uniqueness of name in case of deferred planning.- Parameters:
dag
- the workflow so far.site
- the execution pool for which the create directory job is responsible.- Returns:
- String corresponding to the name of the job.
-
getCleanupJobname
protected java.lang.String getCleanupJobname(ADag dag, java.lang.String site)
It returns the name of the untar job, that is to be assigned. The name takes into account the workflow name while constructing it, as that is thing that can guarentee uniqueness of name in case of deferred planning.- Parameters:
dag
- the workflow so far.site
- the execution pool for which the create directory job is responsible.- Returns:
- String corresponding to the name of the job.
-
makeUntarJob
protected Job makeUntarJob(java.lang.String site, java.lang.String jobName, java.lang.String wpBasename)
It creates a untar job , that untars the worker package that is staged by the setup transfer job.- Parameters:
site
- the execution pool for which the create dir job is to be created.jobName
- the name that is to be assigned to the job.wpBasename
- the basename of the worker package that is staged to remote site.- Returns:
- create dir job.
-
addDefaultTCEntry
private TransformationCatalogEntry addDefaultTCEntry(java.lang.String site, java.lang.String pegasusHome, SysInfo sysinfo, boolean useFullPath, java.lang.String name, java.lang.String executable)
Returns a default TC entry to be used in case entry is not found in the transformation catalog. It also attempts to add the transformation catalog entry to the underlying TC store.- Parameters:
site
- the site for which the default entry is required.pegasusHome
- the path to deployed worker packagesysinfo
- the system information of that site.useFullPath
- boolean indicating whether to use just the basename or the full pathname
- the logical name of the transformationexecutable
- the basename of the executable- Returns:
- the default entry.
-
addDefaultTCEntryForPegasusWebsite
protected TransformationCatalogEntry addDefaultTCEntryForPegasusWebsite(java.lang.String site, java.lang.String name)
Returns a default TC entry for the pegasus site. The entry points to the http webserver on the pegasus website. It also attempts to add the transformation catalog entry to the TC store.- Parameters:
site
- the execution site for which we need a matching static binary.name
- logical name of the transformation- Returns:
- the default entry.
-
constructDefaultURLToPegasusWorkerPackage
protected java.lang.String constructDefaultURLToPegasusWorkerPackage(java.lang.String name, SysInfo sysinfo)
Constructs the default URL's for the pegasus worker package. If the user has not specified the URL to the source directory in Pegaus Properties then the URL constructed points to the pegasus website. The version of Pegasus retrieved is the one against which the planner is executing.- Parameters:
name
- the logical name of the executable, usually worker|binary.sysinfo
- the sysinfo for which the path is required.- Returns:
- url
-
defaultUntarTCEntry
private TransformationCatalogEntry defaultUntarTCEntry(SiteCatalogEntry site)
Returns a default TC entry to be used in case entry is not found in the transformation catalog.- Parameters:
site
- the site for which the default entry is required.- Returns:
- the default entry.
-
getBasename
protected java.lang.String getBasename(java.lang.String url)
Returns the basename of the URL using substring.- Parameters:
url
-- Returns:
- basename
-
getStagingSite
private java.lang.String getStagingSite(java.lang.String site)
Returns the staging site for a particular execution site. If worker node execution is enabled, then the staging site is the submit directory for the workflow on the local sit.e- Parameters:
site
- the execution site.- Returns:
- the staging site
-
-