Class T2
- java.lang.Object
-
- edu.isi.pegasus.planner.transfer.implementation.Abstract
-
- edu.isi.pegasus.planner.transfer.implementation.AbstractMultipleFTPerXFERJob
-
- edu.isi.pegasus.planner.transfer.implementation.T2
-
- All Implemented Interfaces:
Implementation
,MultipleFTPerXFERJob
public class T2 extends AbstractMultipleFTPerXFERJob
The implementation that creates transfer jobs referring to the T2 executable distributed with the Pegasus. T2 extends upon the multiple transfers provided by the transfer executable, providing for conditional/optional transfers, and retry in case of url if alternative source or destinations are specified.The T2 client is generally invoked on the remote execution sites, unless the user uses the thirdparty transfer option, in which case the T2 is invoked on the submit host. Hence there should be an entry in the transformation catalog for logical transformation
T2
at the execution sites. T2 is distributed as part of the Pegasus worker package and can be found at $PEGASUS_HOME/bin/T2.It leads to the creation of the setup chmod jobs to the workflow, that appear as parents to compute jobs in case the transfer implementation does not preserve the X bit on the file being transferred. This is required for staging of executables as part of the workflow. The setup jobs are only added as children to the stage in jobs.
In order to use the transfer implementation implemented by this class,
- the property pegasus.transfer.*.impl must be set to value T2.
There should be an entry in the transformation catalog with the fully qualified name as
pegasus::T2
for all the sites where workflow is run, or on the local site in case of third party transfers.The arguments with which the client is invoked can be specified
- by specifying the property pegasus.transfer.arguments - associating the Pegasus profile key transfer.arguments
- Version:
- $Revision$
- Author:
- Karan Vahi
-
-
Field Summary
Fields Modifier and Type Field Description static java.lang.String
DERIVATION_NAME
The name of the underlying derivation.static java.lang.String
DERIVATION_NAMESPACE
The derivation namespace for for the transfer job.static java.lang.String
DERIVATION_VERSION
The derivation version number for the transfer job.static java.lang.String
DESCRIPTION
A short description of the transfer implementation.protected java.lang.String
mNumOfTXProcesses
The number of g-u-c processes that are spawned to transfer the files in one transfer job.protected java.lang.String
mNumOfTXStreams
The number of streams that each g-u-c process opens to do the ftp transfer.protected boolean
mUseForce
Whether to use force option for the transfer executable or not.static java.lang.String
TRANSFORMATION_NAME
The name of the underlying transformation that is queried for in the Transformation Catalog.static java.lang.String
TRANSFORMATION_NAMESPACE
The transformation namespace for the transfer job.static java.lang.String
TRANSFORMATION_VERSION
The version number for the transfer job.-
Fields inherited from class edu.isi.pegasus.planner.transfer.implementation.Abstract
CHANGE_XBIT_TRANSFORMATION, mAddNodesForSettingXBit, mChmodDisabledForAllSites, mDisabledChmodSites, mLocalIrodsEnv, mLocalIrodsEnvBasename, mLocalS3cfg, mLocalS3cfgBasename, mLocalUserProxy, mLocalUserProxyBasename, mLogger, mPOptions, mProps, mRefiner, mSiteStore, mTCHandle, NOOP_PREFIX, SET_XBIT_PREFIX, XBIT_DERIVATION_NS, XBIT_DERIVATION_VERSION, XBIT_EXECUTABLE_BASENAME, XBIT_TRANSFORMATION_NS, XBIT_TRANSFORMATION_VERSION
-
Fields inherited from interface edu.isi.pegasus.planner.transfer.Implementation
TRANSFER_UNIVERSE, VERSION
-
-
Constructor Summary
Constructors Constructor Description T2(PegasusBag bag)
The overloaded constructor, that is called by the Factory to load the class.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description boolean
doesPreserveXBit()
Returns a boolean indicating whether the transfer protocol being used by the implementation preserves the X Bit or not while staging.protected java.lang.String
generateArgumentString(TransferJob job)
It constructs the arguments to the transfer executable that need to be passed to the executable referred to in this transfer mode.protected java.lang.String
getCompleteTCName()
Returns the complete name for the transformation.protected java.lang.String
getDerivationName()
Returns the logical name of the derivation that this implementation refers to.protected java.lang.String
getDerivationNamespace()
Returns the namespace of the derivation that this implementation refers to.protected java.lang.String
getDerivationVersion()
Returns the version of the derivation that this implementation refers to.java.lang.String
getDescription()
Returns a textual description of the transfer implementation.protected java.util.List
getEnvironmentVariables(java.lang.String site)
Returns the environment profiles that are required for the default entry to sensibly work.TransformationCatalogEntry
getTransformationCatalogEntry(java.lang.String siteHandle, int jobClass)
Retrieves the transformation catalog entry for the executable that is being used to transfer the files in the implementation.boolean
useThirdPartyTransferAlways()
Return a boolean indicating whether the transfers to be done always in a third party transfer mode.protected void
writeStdInAndAssociateCredentials(TransferJob job, java.io.FileWriter writer, java.util.Collection files, java.lang.String stagingSite, int jobClass)
Writes to a FileWriter stream the stdin which T2 takes via standard input.-
Methods inherited from class edu.isi.pegasus.planner.transfer.implementation.AbstractMultipleFTPerXFERJob
createTransferJob, defaultTCEntry, postProcess, prepareSTDINAndAssociateCredentials
-
Methods inherited from class edu.isi.pegasus.planner.transfer.implementation.Abstract
addSetXBitJobs, addSetXBitJobs, addSetXBitJobs, applyPriority, checkAndTransferIrodsEnvFile, checkAndTransferProxy, construct, createNoOPJob, createSetXBitJob, createSetXBitJob, createSetXBitJob, determineDisabledChmodSites, disableChmodJobCreation, getNOOPJobName, getPriority, getSetXBitJobName, setRefiner
-
-
-
-
Field Detail
-
TRANSFORMATION_NAMESPACE
public static final java.lang.String TRANSFORMATION_NAMESPACE
The transformation namespace for the transfer job.- See Also:
- Constant Field Values
-
TRANSFORMATION_NAME
public static final java.lang.String TRANSFORMATION_NAME
The name of the underlying transformation that is queried for in the Transformation Catalog.- See Also:
- Constant Field Values
-
TRANSFORMATION_VERSION
public static final java.lang.String TRANSFORMATION_VERSION
The version number for the transfer job.
-
DERIVATION_NAMESPACE
public static final java.lang.String DERIVATION_NAMESPACE
The derivation namespace for for the transfer job.- See Also:
- Constant Field Values
-
DERIVATION_NAME
public static final java.lang.String DERIVATION_NAME
The name of the underlying derivation.- See Also:
- Constant Field Values
-
DERIVATION_VERSION
public static final java.lang.String DERIVATION_VERSION
The derivation version number for the transfer job.- See Also:
- Constant Field Values
-
DESCRIPTION
public static final java.lang.String DESCRIPTION
A short description of the transfer implementation.- See Also:
- Constant Field Values
-
mNumOfTXProcesses
protected java.lang.String mNumOfTXProcesses
The number of g-u-c processes that are spawned to transfer the files in one transfer job.
-
mNumOfTXStreams
protected java.lang.String mNumOfTXStreams
The number of streams that each g-u-c process opens to do the ftp transfer.
-
mUseForce
protected boolean mUseForce
Whether to use force option for the transfer executable or not.
-
-
Constructor Detail
-
T2
public T2(PegasusBag bag)
The overloaded constructor, that is called by the Factory to load the class.- Parameters:
bag
- the bag of initialization objects.
-
-
Method Detail
-
useThirdPartyTransferAlways
public boolean useThirdPartyTransferAlways()
Return a boolean indicating whether the transfers to be done always in a third party transfer mode. A value of false, results in the direct or peer to peer transfers being done.A value of false does not preclude third party transfers. They still can be done, by setting the property "pegasus.transfer.*.thirdparty.sites".
- Returns:
- boolean indicating whether to always use third party transfers or not.
- See Also:
PegasusProperties.getThirdPartySites(String)
-
doesPreserveXBit
public boolean doesPreserveXBit()
Returns a boolean indicating whether the transfer protocol being used by the implementation preserves the X Bit or not while staging.- Returns:
- boolean
-
getDescription
public java.lang.String getDescription()
Returns a textual description of the transfer implementation.- Returns:
- a short textual description
-
getTransformationCatalogEntry
public TransformationCatalogEntry getTransformationCatalogEntry(java.lang.String siteHandle, int jobClass)
Retrieves the transformation catalog entry for the executable that is being used to transfer the files in the implementation.- Parameters:
siteHandle
- the handle of the site where the transformation is to be searched.jobClass
- the job Class for the newly added job. Can be one of the following: stage-in stage-out inter-pool transfer stage-in worker transfer- Returns:
- the transformation catalog entry if found, else null.
-
getDerivationNamespace
protected java.lang.String getDerivationNamespace()
Returns the namespace of the derivation that this implementation refers to.- Specified by:
getDerivationNamespace
in classAbstractMultipleFTPerXFERJob
- Returns:
- the namespace of the derivation.
-
getDerivationName
protected java.lang.String getDerivationName()
Returns the logical name of the derivation that this implementation refers to.- Specified by:
getDerivationName
in classAbstractMultipleFTPerXFERJob
- Returns:
- the name of the derivation.
-
getDerivationVersion
protected java.lang.String getDerivationVersion()
Returns the version of the derivation that this implementation refers to.- Specified by:
getDerivationVersion
in classAbstractMultipleFTPerXFERJob
- Returns:
- the version of the derivation.
-
generateArgumentString
protected java.lang.String generateArgumentString(TransferJob job)
It constructs the arguments to the transfer executable that need to be passed to the executable referred to in this transfer mode.- Specified by:
generateArgumentString
in classAbstractMultipleFTPerXFERJob
- Parameters:
job
- the object containing the transfer node.- Returns:
- the argument string
-
writeStdInAndAssociateCredentials
protected void writeStdInAndAssociateCredentials(TransferJob job, java.io.FileWriter writer, java.util.Collection files, java.lang.String stagingSite, int jobClass) throws java.lang.Exception
Writes to a FileWriter stream the stdin which T2 takes via standard input.- Specified by:
writeStdInAndAssociateCredentials
in classAbstractMultipleFTPerXFERJob
- Parameters:
job
- the transfer job.writer
- the writer to the stdin file.files
- Collection ofFileTransfer
objects containing the information about sourceam fin and destURL's.stagingSite
- the site where the data will be populated by first level staging jobs.jobClass
- the job Class for the newly added job. Can be one of the following: stage-in stage-out inter-pool transfer- Throws:
java.lang.Exception
- See Also:
org.griphyn.cPlanner.classes.FileTransfer#toString()
-
getCompleteTCName
protected java.lang.String getCompleteTCName()
Returns the complete name for the transformation.- Specified by:
getCompleteTCName
in classAbstractMultipleFTPerXFERJob
- Returns:
- the complete name.
-
getEnvironmentVariables
protected java.util.List getEnvironmentVariables(java.lang.String site)
Returns the environment profiles that are required for the default entry to sensibly work.- Specified by:
getEnvironmentVariables
in classAbstractMultipleFTPerXFERJob
- Parameters:
site
- the site where the job is going to run.- Returns:
- List of environment variables, else null in case where the required environment variables could not be found.
-
-