Class Bundle
- java.lang.Object
-
- edu.isi.pegasus.planner.transfer.AbstractRefiner
-
- edu.isi.pegasus.planner.transfer.MultipleFTPerXFERJobRefiner
-
- edu.isi.pegasus.planner.transfer.refiner.Basic
-
- edu.isi.pegasus.planner.transfer.refiner.Bundle
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description protected class
Bundle.BundleValue
protected class
Bundle.PoolTransfer
A container to store the transfers that need to be done on a single pool.protected class
Bundle.TransferContainer
A container class for storing the name of the transfer job, the list of file transfers that the job is responsible for.
-
Field Summary
Fields Modifier and Type Field Description static java.lang.String
DEFAULT_LOCAL_STAGE_IN_BUNDLE_FACTOR
The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for stageing in data for the workflow.static java.lang.String
DEFAULT_LOCAL_STAGE_OUT_BUNDLE_FACTOR
The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for stageing out data for the workflow.static java.lang.String
DEFAULT_REMOTE_STAGE_IN_BUNDLE_FACTOR
The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for stageing in data for the workflow.static java.lang.String
DEFAULT_REMOTE_STAGE_OUT_BUNDLE_FACTOR
The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for stageing out data for the workflow.static java.lang.String
DESCRIPTION
A short description of the transfer refinement.protected boolean
mAddNodesForSettingXBit
A boolean indicating whether chmod jobs should be created that set the xbit in case of executable staging.private int
mCurrentSOLevel
The current level of the jobs being traversed.protected java.lang.String
mJobPrefix
The job prefix that needs to be applied to the job file basenames.protected Pegasus
mPegasusProfilesInProperties
Pegasus Profiles specified in the properties.private ReplicaCatalogBridge
mRCB
The handle to the replica catalog bridge.private java.util.Map
mRelationsParentMap
The map indexed by compute jobnames that contains the list of stagin job names that are being added during the traversal of the workflow.protected java.util.Map
mSetupMap
The map indexed by staged executable logical name.protected SiteStore
mSiteStore
Handle to the SiteStoreprotected Bundle.BundleValue
mStageinLocalBundleValue
The BundleValue that evaluates for local stage in jobs.private java.util.Map
mStageInLocalMap
The map containing the list of stage in transfer jobs that are being created for the workflow indexed by the execution poolname.protected Bundle.BundleValue
mStageInRemoteBundleValue
The BundleValue that evaluates for remote stage-in jobs.private java.util.Map
mStageInRemoteMap
The map containing the list of stage in transfer jobs that are being created for the workflow indexed by the execution poolname.protected Bundle.BundleValue
mStageOutLocalBundleValue
The BundleValue that evaluates for local stage out jobs.private java.util.Map<java.lang.String,Bundle.PoolTransfer>
mStageOutLocalMapPerLevel
A map indexed by site name, that contains the pointer to the local stage out PoolTransfer objects for that site.protected Bundle.BundleValue
mStageOutRemoteBundleValue
The BundleValue that evaluates for remote stage out jobs.private java.util.Map<java.lang.String,Bundle.PoolTransfer>
mStageOutRemoteMapPerLevel
A map indexed by site name, that contains the pointer to the remote stage out PoolTransfer objects for that site.-
Fields inherited from class edu.isi.pegasus.planner.transfer.refiner.Basic
mCreateRegistrationJobs, mFileTable, mLogMsg, mPPS, mRelationsMap
-
Fields inherited from class edu.isi.pegasus.planner.transfer.AbstractRefiner
mDAG, mLogger, mPOptions, mProps, mRemoteTransfers, mTPT, mTXInterImplementation, mTXStageInImplementation, mTXStageOutImplementation, mTXSymbolicLinkImplementation, mXMLStore
-
Fields inherited from interface edu.isi.pegasus.planner.transfer.Refiner
INTER_POOL_PREFIX, LOCAL_PREFIX, REGISTER_PREFIX, REMOTE_PREFIX, STAGE_IN_PREFIX, STAGE_OUT_PREFIX
-
-
Constructor Summary
Constructors Constructor Description Bundle(ADag dag, PegasusBag bag)
The overloaded constructor.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
addStageInXFERNodes(Job job, boolean localTransfer, java.util.Collection files, int type, java.util.Map<java.lang.String,Bundle.PoolTransfer> stageInMap, Bundle.BundleValue bundleValue, Implementation implementation)
Adds the stage in transfer nodes which transfer the input files for a job, from the location returned from the replica catalog to the job's execution pool.void
addStageInXFERNodes(Job job, java.util.Collection<FileTransfer> files, java.util.Collection<FileTransfer> symlinkFiles)
Adds the stage in transfer nodes which transfer the input files for a job, from the location returned from the replica catalog to the job's execution pool.void
addStageOutXFERNodes(Job job, java.util.Collection files, ReplicaCatalogBridge rcb, boolean localTransfer, boolean deletedLeaf)
Adds the stageout transfer nodes, that stage data to an output site specified by the user.void
done()
Signals that the traversal of the workflow is done.void
doneStageIn(java.util.Map<java.lang.String,Bundle.PoolTransfer> stageInMap, Implementation implementation, int stageInJobType, boolean localTransfer)
Signals that the traversal of the workflow is done.protected java.lang.String
getComputeJobBundleValue(Job job)
Returns the bundle value associated with a compute job as a String.protected java.lang.String
getDefaultBundleValueFromProperties(java.lang.String key, java.lang.String defaultKey, java.lang.String defaultValue)
Returns the default value for the clustering/bundling of jobs to be used.java.lang.String
getDescription()
Returns a textual description of the transfer mode.Bundle.PoolTransfer
getStageOutPoolTransfer(java.lang.String site, boolean localTransfer, int num)
Returns the appropriate pool transfer for a particular site.protected void
initializeBundleValues()
Initializes the bundle value variables, that are responsible determining the bundle values.protected java.util.Map
resetStageOutMap(java.util.Map<java.lang.String,Bundle.PoolTransfer> map, boolean localTransfer)
Resets a single mapprotected void
resetStageOutMaps()
Resets the local and remote stage out maps.-
Methods inherited from class edu.isi.pegasus.planner.transfer.refiner.Basic
addInterSiteTXNodes, addJob, addRelation, addRelation, addStageInXFERNodes, addStageOutXFERNodes, appendAttribute, constructFileKey, createRegistrationJob, getJobPriority, logRefinerAction
-
Methods inherited from class edu.isi.pegasus.planner.transfer.MultipleFTPerXFERJobRefiner
loadImplementations
-
Methods inherited from class edu.isi.pegasus.planner.transfer.AbstractRefiner
addStageInXFERNodes, getWorkflow, getXMLProducer, isSiteThirdParty, logConfigMessages, refinerPreferenceForLocalTransferJobs, refinerPreferenceForTransferJobLocation, runTPTOnRemoteSite, runTransferRemotely
-
-
-
-
Field Detail
-
DESCRIPTION
public static final java.lang.String DESCRIPTION
A short description of the transfer refinement.- See Also:
- Constant Field Values
-
DEFAULT_LOCAL_STAGE_IN_BUNDLE_FACTOR
public static final java.lang.String DEFAULT_LOCAL_STAGE_IN_BUNDLE_FACTOR
The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for stageing in data for the workflow.- See Also:
- Constant Field Values
-
DEFAULT_REMOTE_STAGE_IN_BUNDLE_FACTOR
public static final java.lang.String DEFAULT_REMOTE_STAGE_IN_BUNDLE_FACTOR
The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for stageing in data for the workflow.- See Also:
- Constant Field Values
-
DEFAULT_LOCAL_STAGE_OUT_BUNDLE_FACTOR
public static final java.lang.String DEFAULT_LOCAL_STAGE_OUT_BUNDLE_FACTOR
The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for stageing out data for the workflow.- See Also:
- Constant Field Values
-
DEFAULT_REMOTE_STAGE_OUT_BUNDLE_FACTOR
public static final java.lang.String DEFAULT_REMOTE_STAGE_OUT_BUNDLE_FACTOR
The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for stageing out data for the workflow.- See Also:
- Constant Field Values
-
mStageInLocalMap
private java.util.Map mStageInLocalMap
The map containing the list of stage in transfer jobs that are being created for the workflow indexed by the execution poolname.
-
mStageInRemoteMap
private java.util.Map mStageInRemoteMap
The map containing the list of stage in transfer jobs that are being created for the workflow indexed by the execution poolname.
-
mRelationsParentMap
private java.util.Map mRelationsParentMap
The map indexed by compute jobnames that contains the list of stagin job names that are being added during the traversal of the workflow. This is used to construct the relations that need to be added to workflow, once the traversal is done.
-
mStageinLocalBundleValue
protected Bundle.BundleValue mStageinLocalBundleValue
The BundleValue that evaluates for local stage in jobs.
-
mStageInRemoteBundleValue
protected Bundle.BundleValue mStageInRemoteBundleValue
The BundleValue that evaluates for remote stage-in jobs.
-
mStageOutLocalBundleValue
protected Bundle.BundleValue mStageOutLocalBundleValue
The BundleValue that evaluates for local stage out jobs.
-
mStageOutRemoteBundleValue
protected Bundle.BundleValue mStageOutRemoteBundleValue
The BundleValue that evaluates for remote stage out jobs.
-
mSetupMap
protected java.util.Map mSetupMap
The map indexed by staged executable logical name. Each entry is the name of the corresponding setup job, that changes the XBit on the staged file.
-
mStageOutLocalMapPerLevel
private java.util.Map<java.lang.String,Bundle.PoolTransfer> mStageOutLocalMapPerLevel
A map indexed by site name, that contains the pointer to the local stage out PoolTransfer objects for that site. This is per level of the workflow.
-
mStageOutRemoteMapPerLevel
private java.util.Map<java.lang.String,Bundle.PoolTransfer> mStageOutRemoteMapPerLevel
A map indexed by site name, that contains the pointer to the remote stage out PoolTransfer objects for that site. This is per level of the workflow.
-
mCurrentSOLevel
private int mCurrentSOLevel
The current level of the jobs being traversed.
-
mRCB
private ReplicaCatalogBridge mRCB
The handle to the replica catalog bridge.
-
mJobPrefix
protected java.lang.String mJobPrefix
The job prefix that needs to be applied to the job file basenames.
-
mPegasusProfilesInProperties
protected Pegasus mPegasusProfilesInProperties
Pegasus Profiles specified in the properties.
-
mSiteStore
protected SiteStore mSiteStore
Handle to the SiteStore
-
mAddNodesForSettingXBit
protected boolean mAddNodesForSettingXBit
A boolean indicating whether chmod jobs should be created that set the xbit in case of executable staging.
-
-
Constructor Detail
-
Bundle
public Bundle(ADag dag, PegasusBag bag)
The overloaded constructor.- Parameters:
dag
- the workflow to which transfer nodes need to be added.bag
- the bag of initialization objects
-
-
Method Detail
-
initializeBundleValues
protected void initializeBundleValues()
Initializes the bundle value variables, that are responsible determining the bundle values.
-
getDefaultBundleValueFromProperties
protected java.lang.String getDefaultBundleValueFromProperties(java.lang.String key, java.lang.String defaultKey, java.lang.String defaultValue)
Returns the default value for the clustering/bundling of jobs to be used. The factor is computed by looking up the pegasus profiles in the properties.return value of pegasus profile key if it exists, else return value of pegasus profile defaultKey if it exists, else the defaultValue
- Parameters:
key
- the pegasus profile keydefaultKey
- the default pegasus profile keydefaultValue
- the default value.- Returns:
- the value as string.
-
addStageInXFERNodes
public void addStageInXFERNodes(Job job, java.util.Collection<FileTransfer> files, java.util.Collection<FileTransfer> symlinkFiles)
Adds the stage in transfer nodes which transfer the input files for a job, from the location returned from the replica catalog to the job's execution pool.- Specified by:
addStageInXFERNodes
in interfaceRefiner
- Overrides:
addStageInXFERNodes
in classBasic
- Parameters:
job
-Job
object corresponding to the node to which the files are to be transferred to.files
- Collection ofFileTransfer
objects containing the information about source and destURL's.symlinkFiles
- Collection ofFileTransfer
objects containing source and destination file url's for symbolic linking on compute site.
-
addStageInXFERNodes
public void addStageInXFERNodes(Job job, boolean localTransfer, java.util.Collection files, int type, java.util.Map<java.lang.String,Bundle.PoolTransfer> stageInMap, Bundle.BundleValue bundleValue, Implementation implementation)
Adds the stage in transfer nodes which transfer the input files for a job, from the location returned from the replica catalog to the job's execution pool.- Parameters:
job
-Job
object corresponding to the node to which the files are to be transferred to.localTransfer
- boolean indicating whether transfer has to happen on local site.files
- Collection ofFileTransfer
objects containing the information about source and destURL's.type
- the type of transfer job being createdstageInMap
- Map indexed by site name that gives all the transfers for that site.bundleValue
- used to determine the bundling factor to employ for a job.implementation
- the transfer implementation to use.
-
addStageOutXFERNodes
public void addStageOutXFERNodes(Job job, java.util.Collection files, ReplicaCatalogBridge rcb, boolean localTransfer, boolean deletedLeaf)
Adds the stageout transfer nodes, that stage data to an output site specified by the user.- Specified by:
addStageOutXFERNodes
in interfaceRefiner
- Overrides:
addStageOutXFERNodes
in classBasic
- Parameters:
job
-Job
object corresponding to the node to which the files are to be transferred to.files
- Collection ofFileTransfer
objects containing the information about source and destURL's.rcb
- bridge to the Replica Catalog. Used for creating registration nodes in the workflow.localTransfer
- whether the transfer should be on local site or not.deletedLeaf
- to specify whether the node is being added for a deleted node by the reduction engine or not. default: false
-
getComputeJobBundleValue
protected java.lang.String getComputeJobBundleValue(Job job)
Returns the bundle value associated with a compute job as a String.- Parameters:
job
-- Returns:
- value as String or NULL
-
done
public void done()
Signals that the traversal of the workflow is done. At this point the transfer nodes are actually constructed traversing through the transfer containers and the stdin of the transfer jobs written.
-
doneStageIn
public void doneStageIn(java.util.Map<java.lang.String,Bundle.PoolTransfer> stageInMap, Implementation implementation, int stageInJobType, boolean localTransfer)
Signals that the traversal of the workflow is done. At this point the transfer nodes are actually constructed traversing through the transfer containers and the stdin of the transfer jobs written.- Parameters:
stageInMap
- maps site names to PoolTransferimplementation
- the transfer implementation to usestageInJobType
- whether a stagein or symlink stagein joblocalTransfer
- indicates whether transfer job needs to run on local site or not.
-
getDescription
public java.lang.String getDescription()
Returns a textual description of the transfer mode.- Specified by:
getDescription
in interfaceRefiner
- Overrides:
getDescription
in classBasic
- Returns:
- a short textual description
-
getStageOutPoolTransfer
public Bundle.PoolTransfer getStageOutPoolTransfer(java.lang.String site, boolean localTransfer, int num)
Returns the appropriate pool transfer for a particular site.- Parameters:
site
- the site for which the PT is reqd.localTransfer
- whethe the associated transfer job runs on local site or remote.num
- the number of Stageout jobs required for that Pool.- Returns:
- the PoolTransfer
-
resetStageOutMaps
protected void resetStageOutMaps()
Resets the local and remote stage out maps.
-
resetStageOutMap
protected java.util.Map resetStageOutMap(java.util.Map<java.lang.String,Bundle.PoolTransfer> map, boolean localTransfer)
Resets a single map- Parameters:
map
- the map to be resetlocalTransfer
- whether the transfer jobs need to run on local site or not- Returns:
- the reset map
-
-