Class CondorGenerator
- java.lang.Object
-
- edu.isi.pegasus.planner.code.generator.Abstract
-
- edu.isi.pegasus.planner.code.generator.condor.CondorGenerator
-
- All Implemented Interfaces:
CodeGenerator
- Direct Known Subclasses:
HashedFile
public class CondorGenerator extends Abstract
This class generates the condor submit files for the DAG which has to be submitted to the Condor DagMan.- Version:
- $Revision$
- Author:
- Gaurang Mehta, Karan Vahi
-
-
Field Summary
Fields Modifier and Type Field Description static java.lang.String
CONDOR_DAGMAN_LOGICAL_NAME
The logical name with which to query the transformation catalog for the condor_dagman executable, that ends up running the mini dag as one job.static java.lang.String
CONDOR_DAGMAN_NAMESPACE
The namespace to use for condor dagman.static java.lang.String
DAGMAN_PROPERTIES_PREFIX
The prefix for DAGMan specific propertiesstatic int
DEFAULT_CHMOD_PRIORITY_KEY
The default priority key associated with chmod jobs.static int
DEFAULT_CLEANUP_PRIORITY_KEY
The default priority key associated with the cleanup jobs.static int
DEFAULT_CREATE_DIR_PRIORITY_KEY
The default priority key associated with the create dir jobs.static int
DEFAULT_INTER_SITE_PRIORITY_KEY
The default priority key associated with the inter site transfer jobs.static java.lang.String
DEFAULT_PERIODIC_RELEASE_VALUE
Default value for the periodic_release for a jobstatic java.lang.String
DEFAULT_PERIODIC_REMOVE_VALUE
Default value for the periodic_remove for a jobstatic int
DEFAULT_REPLICA_REG_PRIORITY_KEY
The default priority key associated with the replica registration jobs.static int
DEFAULT_STAGE_IN_PRIORITY_KEY
The default priority key associated with the stagein jobs.static int
DEFAULT_STAGE_OUT_PRIORITY_KEY
The default priority key associated with the stage out jobs.static java.lang.String
DEFAULT_SUBDAG_CATEGORY_KEY
The default category for the sub dax jobs.private boolean
mAssignDefaultJobPriorities
Boolean indicating whether to assign job priorities or not.protected ADag
mConcreteWorkflow
The workflow for which the code has to be generated.private long
mCondorVersion
The long value of condor version.protected java.io.PrintWriter
mDagWriter
The file handle to the .dag file.protected boolean
mDone
A boolean indicating whether the files have been generated or not.protected GridStartFactory
mGridStartFactory
The handle to the GridStart Factory.protected boolean
mInitializeGridStart
A boolean indicating whether grid start has been initialized or not.protected java.lang.String
mPoolClass
Specifies the implementing class for the pool interface.static java.lang.String
mSeparator
The nice separator, define once, use often.private SiteStore
mSiteStore
The handle to the site catalog store.protected CondorStyleFactory
mStyleFactory
Handle to the Style factory, that is used for this workflow.protected TransformationCatalog
mTCHandle
Handle to the Transformation Catalog.protected java.lang.String
mTempLogFile
The name of the log file in the /tmp directory-
Fields inherited from class edu.isi.pegasus.planner.code.generator.Abstract
mBag, mLogger, mPOptions, mProps, mSubmitFileDir
-
Fields inherited from interface edu.isi.pegasus.planner.code.CodeGenerator
VERSION
-
-
Constructor Summary
Constructors Constructor Description CondorGenerator()
The default constructor.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description protected void
applyStyle(Job job, java.io.PrintWriter writer)
Applies a submit file style to the job, according to the fact whether the job has to be submitted directly to condor or to a remote jobmanager via CondorG and GRAM.protected void
concreteDagEmpty(java.lang.String filename, ADag dag)
A callout method that dictates what needs to be done in case the concrete plan that is generated is empty.protected Job
constructDAGJob(java.lang.String name, java.lang.String directory, java.lang.String dagBasename)
Constructs a job that plans and submits the partitioned workflow, referred to by a Partition.private java.util.Map
constructMap(java.lang.String propValue)
Constructs a map with the numbers/values to be passed in the RSL handle for certain pools.private TransformationCatalogEntry
constructTCEntryFromEnvironment()
Returns a tranformation catalog entry object constructed from the environment An entry is constructed if either of the following environment variables are defined 1) CONDOR_HOME 2) CONDOR_LOCATION CONDOR_HOME takes precedence over CONDOR_LOCATIONprivate TransformationCatalogEntry
constructTCEntryFromEnvProfiles(ENV env)
Returns a tranformation catalog entry object constructed from the environment An entry is constructed if either of the following environment variables are defined 1) CONDOR_HOME 2) CONDOR_LOCATION CONDOR_HOME takes precedence over CONDOR_LOCATIONprivate TransformationCatalogEntry
defaultTCEntry(java.lang.String site)
Returns a default TC entry to be used in case entry is not found in the transformation catalog.java.util.Collection<java.io.File>
generateCode(ADag dag)
Generates the code for the concrete workflow in Condor DAGMAN and CondorG input format.void
generateCode(ADag dag, Job job)
Generates the code (condor submit file) for a single job.protected boolean
generateLogFileSymlink(java.lang.String logFile, java.lang.String symlink)
This method generates a symlink to the actual log file written in the local temp directory.java.util.Map<java.lang.String,java.lang.String>
getAdditionalBraindumpEntries(ADag workflow)
Returns a Map containing additional braindump entries that are specific to a Code Generator.protected java.lang.String
getBasename(java.lang.String prefix, java.lang.String suffix)
A covenience method to construct the basename.protected java.lang.String
getCategoryDAGManKnobs(PegasusProperties properties)
Write out the DAGMan knobs for each category the user mentions in the properties.protected java.lang.String
getCondorLogInSubmitDirectory()
Returns the path to the condor log file in the submit directory.protected java.lang.String
getCondorLogInSubmitDirectory(ADag dag)
Returns the path to the condor log file in the submit directory.protected java.lang.String
getCondorLogInTmpDirectory()
Returns the name of Condor log file in a tmp directory that is created if generation of symlink for condor logs is turned on.protected java.lang.String
getDAGMANOutFilename(ADag dag)
Returns the basename of the file, that contains the output of the dagman while running the dag generated for the workflow.protected int
getJobPriority(Job job, int depth)
Computes the priority for a job based on job type and depth in the workflowprivate java.lang.String
gridstart(java.io.PrintWriter writer, Job job, boolean isGlobusJob)
This function creates the stdio handling with and without gridstart.protected void
handleCondorVarForJob(Job job)
It updates/adds the condor variables that are got through the Dax with the values specified in the properties file, pool config file or adds some variables internally.protected void
handleEnvVarForJob(Job sinfo)
It updates/adds the environment variables that are got through the Dax with the values specified in the properties file, pool config file or adds some variables internally.protected void
handleGlobusRSLForJob(Job sinfo)
It updates/adds the the Globus RSL parameters got through the dax that are in Job object.void
initialize(PegasusBag bag)
Initializes the Code Generator implementation.protected java.io.File
initializeDagFileWriter(java.lang.String filename, ADag workflow)
Initializes the file handler to the dag file and writes the header to it.protected boolean
isInteger(java.lang.String value)
Returns a boolean indicating whether the value represented is an integer or not.void
populatePeriodicReleaseAndRemoveInJob(Job job)
Populates the periodic release and remove values in the job.protected void
printDagString(java.lang.String str)
Writes a string to the dag file.void
reset()
Resets the Code Generator implementation.boolean
startMonitoring()
Starts monitoring of the workflow by invoking a workflow monitor daemon tailstatd.protected void
writeDagFileTail(ADag dag)
It writes the relations making up the DAG in the dag file and and closes the file handle to it.protected void
writeDOTFile(java.lang.String filename, ADag dag)
Writes out the DOT file in the submit directory.protected void
writeJobMapFile(java.lang.String filename, ADag dag)
Writes out the job map file in the submit directory.-
Methods inherited from class edu.isi.pegasus.planner.code.generator.Abstract
getDAGFilename, getDAGFilename, getFileBaseName, getWriter, writeOutBraindump, writeOutDAXReplicaStore, writeOutNotifications, writeOutStampedeEvents
-
-
-
-
Field Detail
-
DEFAULT_SUBDAG_CATEGORY_KEY
public static final java.lang.String DEFAULT_SUBDAG_CATEGORY_KEY
The default category for the sub dax jobs.- See Also:
- Constant Field Values
-
mSeparator
public static final java.lang.String mSeparator
The nice separator, define once, use often.- See Also:
- Constant Field Values
-
DEFAULT_PERIODIC_RELEASE_VALUE
public static final java.lang.String DEFAULT_PERIODIC_RELEASE_VALUE
Default value for the periodic_release for a job- See Also:
- Constant Field Values
-
DEFAULT_PERIODIC_REMOVE_VALUE
public static final java.lang.String DEFAULT_PERIODIC_REMOVE_VALUE
Default value for the periodic_remove for a job- See Also:
- Constant Field Values
-
CONDOR_DAGMAN_NAMESPACE
public static final java.lang.String CONDOR_DAGMAN_NAMESPACE
The namespace to use for condor dagman.- See Also:
- Constant Field Values
-
CONDOR_DAGMAN_LOGICAL_NAME
public static final java.lang.String CONDOR_DAGMAN_LOGICAL_NAME
The logical name with which to query the transformation catalog for the condor_dagman executable, that ends up running the mini dag as one job.- See Also:
- Constant Field Values
-
DAGMAN_PROPERTIES_PREFIX
public static final java.lang.String DAGMAN_PROPERTIES_PREFIX
The prefix for DAGMan specific properties- See Also:
- Constant Field Values
-
DEFAULT_STAGE_IN_PRIORITY_KEY
public static final int DEFAULT_STAGE_IN_PRIORITY_KEY
The default priority key associated with the stagein jobs.- See Also:
- Constant Field Values
-
DEFAULT_INTER_SITE_PRIORITY_KEY
public static final int DEFAULT_INTER_SITE_PRIORITY_KEY
The default priority key associated with the inter site transfer jobs.- See Also:
- Constant Field Values
-
DEFAULT_CREATE_DIR_PRIORITY_KEY
public static final int DEFAULT_CREATE_DIR_PRIORITY_KEY
The default priority key associated with the create dir jobs.- See Also:
- Constant Field Values
-
DEFAULT_CHMOD_PRIORITY_KEY
public static final int DEFAULT_CHMOD_PRIORITY_KEY
The default priority key associated with chmod jobs.- See Also:
- Constant Field Values
-
DEFAULT_STAGE_OUT_PRIORITY_KEY
public static final int DEFAULT_STAGE_OUT_PRIORITY_KEY
The default priority key associated with the stage out jobs.- See Also:
- Constant Field Values
-
DEFAULT_REPLICA_REG_PRIORITY_KEY
public static final int DEFAULT_REPLICA_REG_PRIORITY_KEY
The default priority key associated with the replica registration jobs.- See Also:
- Constant Field Values
-
DEFAULT_CLEANUP_PRIORITY_KEY
public static final int DEFAULT_CLEANUP_PRIORITY_KEY
The default priority key associated with the cleanup jobs.- See Also:
- Constant Field Values
-
mTCHandle
protected TransformationCatalog mTCHandle
Handle to the Transformation Catalog.
-
mSiteStore
private SiteStore mSiteStore
The handle to the site catalog store.
-
mPoolClass
protected java.lang.String mPoolClass
Specifies the implementing class for the pool interface. Contains the name of the class that implements the pool interface the user has asked at runtime.
-
mDagWriter
protected java.io.PrintWriter mDagWriter
The file handle to the .dag file. A part of the dag file is printed as we write the submit files, to insert the appropriate postscripts for handling exit codes.
-
mTempLogFile
protected java.lang.String mTempLogFile
The name of the log file in the /tmp directory
-
mDone
protected boolean mDone
A boolean indicating whether the files have been generated or not.
-
mConcreteWorkflow
protected ADag mConcreteWorkflow
The workflow for which the code has to be generated.
-
mStyleFactory
protected CondorStyleFactory mStyleFactory
Handle to the Style factory, that is used for this workflow.
-
mGridStartFactory
protected GridStartFactory mGridStartFactory
The handle to the GridStart Factory.
-
mInitializeGridStart
protected boolean mInitializeGridStart
A boolean indicating whether grid start has been initialized or not.
-
mCondorVersion
private long mCondorVersion
The long value of condor version.
-
mAssignDefaultJobPriorities
private boolean mAssignDefaultJobPriorities
Boolean indicating whether to assign job priorities or not.
-
-
Method Detail
-
initialize
public void initialize(PegasusBag bag) throws CodeGeneratorException
Initializes the Code Generator implementation. Initializes the various writers.- Specified by:
initialize
in interfaceCodeGenerator
- Overrides:
initialize
in classAbstract
- Parameters:
bag
- the bag of initialization objects.- Throws:
CodeGeneratorException
- in case of any error occuring code generation.
-
generateCode
public java.util.Collection<java.io.File> generateCode(ADag dag) throws CodeGeneratorException
Generates the code for the concrete workflow in Condor DAGMAN and CondorG input format. Returns only the File object for the DAG file that is written out.- Parameters:
dag
- the concrete workflow.- Returns:
- the Collection of
File
objects for the files written out. - Throws:
CodeGeneratorException
- in case of any error occuring code generation.
-
generateCode
public void generateCode(ADag dag, Job job) throws CodeGeneratorException
Generates the code (condor submit file) for a single job.- Parameters:
dag
- the dag of which the job is a part of.job
- theJob
object holding the information about that particular job.- Throws:
CodeGeneratorException
- in case of any error occuring code generation.
-
startMonitoring
public boolean startMonitoring() throws org.griphyn.vdl.euryale.VTorInUseException
Starts monitoring of the workflow by invoking a workflow monitor daemon tailstatd. The tailstatd is picked up from the default path of $PEGASUS_HOME/bin/tailstatd.- Specified by:
startMonitoring
in interfaceCodeGenerator
- Overrides:
startMonitoring
in classAbstract
- Returns:
- boolean indicating whether could successfully start the monitor daemon or not.
- Throws:
org.griphyn.vdl.euryale.VTorInUseException
- in case the method is called before the submit files have been generated.
-
reset
public void reset() throws CodeGeneratorException
Resets the Code Generator implementation.- Specified by:
reset
in interfaceCodeGenerator
- Overrides:
reset
in classAbstract
- Throws:
CodeGeneratorException
- in case of any error occuring code generation.
-
constructMap
private java.util.Map constructMap(java.lang.String propValue)
Constructs a map with the numbers/values to be passed in the RSL handle for certain pools. The user ends up specifying these through the properties file. The value of the property is of the form poolname1=value,poolname2=value....- Parameters:
propValue
- the value of the property got from the properties file.- Returns:
- Map
-
constructDAGJob
protected Job constructDAGJob(java.lang.String name, java.lang.String directory, java.lang.String dagBasename)
Constructs a job that plans and submits the partitioned workflow, referred to by a Partition. The main job itself is a condor dagman job that submits the concrete workflow. The concrete workflow is generated by running the planner in the prescript for the job.- Parameters:
name
- the name to be assigned to the job.directory
- the submit directory where the submit files for the partition should reside. this is where the dag file is createddagBasename
- the basename of the dag file created.- Returns:
- the constructed DAG job.
-
defaultTCEntry
private TransformationCatalogEntry defaultTCEntry(java.lang.String site)
Returns a default TC entry to be used in case entry is not found in the transformation catalog.- Parameters:
site
- the site for which the default entry is required.- Returns:
- the default entry.
-
constructTCEntryFromEnvironment
private TransformationCatalogEntry constructTCEntryFromEnvironment()
Returns a tranformation catalog entry object constructed from the environment An entry is constructed if either of the following environment variables are defined 1) CONDOR_HOME 2) CONDOR_LOCATION CONDOR_HOME takes precedence over CONDOR_LOCATION- Returns:
- the constructed entry else null.
-
constructTCEntryFromEnvProfiles
private TransformationCatalogEntry constructTCEntryFromEnvProfiles(ENV env)
Returns a tranformation catalog entry object constructed from the environment An entry is constructed if either of the following environment variables are defined 1) CONDOR_HOME 2) CONDOR_LOCATION CONDOR_HOME takes precedence over CONDOR_LOCATION- Parameters:
env
- the environment profiles.- Returns:
- the entry constructed else null if environment variables not defined.
-
getBasename
protected java.lang.String getBasename(java.lang.String prefix, java.lang.String suffix)
A covenience method to construct the basename.- Parameters:
prefix
- the first half of basenamesuffix
- the latter half of basename- Returns:
- basename
-
initializeDagFileWriter
protected java.io.File initializeDagFileWriter(java.lang.String filename, ADag workflow) throws CodeGeneratorException
Initializes the file handler to the dag file and writes the header to it.- Parameters:
filename
- basename of dag file to be written.dag
- the workflow- Returns:
- the File object for the DAG file.
- Throws:
CodeGeneratorException
- in case of any error occuring code generation.
-
getCategoryDAGManKnobs
protected java.lang.String getCategoryDAGManKnobs(PegasusProperties properties)
Write out the DAGMan knobs for each category the user mentions in the properties.- Parameters:
properties
- the pegasus properties- Returns:
- the String
-
writeDOTFile
protected void writeDOTFile(java.lang.String filename, ADag dag) throws CodeGeneratorException
Writes out the DOT file in the submit directory.- Parameters:
filename
- basename of dot file to be written .dag
- theADag
object.- Throws:
CodeGeneratorException
- in case of any error occuring code generation.
-
writeJobMapFile
protected void writeJobMapFile(java.lang.String filename, ADag dag) throws CodeGeneratorException
Writes out the job map file in the submit directory.- Parameters:
filename
- basename of dot file to be written .dag
- theADag
object.- Throws:
CodeGeneratorException
- in case of any error occuring code generation.
-
writeDagFileTail
protected void writeDagFileTail(ADag dag) throws CodeGeneratorException
It writes the relations making up the DAG in the dag file and and closes the file handle to it.- Parameters:
dag
- the executable workflow- Throws:
CodeGeneratorException
-
printDagString
protected void printDagString(java.lang.String str) throws CodeGeneratorException
Writes a string to the dag file. When calling this function the file handle to file is already initialized.- Parameters:
str
- The String to be printed to the dag file.- Throws:
CodeGeneratorException
-
getCondorLogInTmpDirectory
protected java.lang.String getCondorLogInTmpDirectory()
Returns the name of Condor log file in a tmp directory that is created if generation of symlink for condor logs is turned on.- Returns:
- the name of the log file.
-
getCondorLogInSubmitDirectory
protected java.lang.String getCondorLogInSubmitDirectory()
Returns the path to the condor log file in the submit directory. It can be a symlink.- Parameters:
dag
- the concrete workflow.- Returns:
- the path to condor log file in the submit directory.
-
getCondorLogInSubmitDirectory
protected java.lang.String getCondorLogInSubmitDirectory(ADag dag)
Returns the path to the condor log file in the submit directory. It can be a symlink.- Parameters:
dag
- the concrete workflow.- Returns:
- the path to condor log file in the submit directory.
-
getAdditionalBraindumpEntries
public java.util.Map<java.lang.String,java.lang.String> getAdditionalBraindumpEntries(ADag workflow)
Returns a Map containing additional braindump entries that are specific to a Code Generator.- Specified by:
getAdditionalBraindumpEntries
in classAbstract
- Parameters:
workflow
- the executable workflow- Returns:
- Map containing entries for dag and condor_log
-
generateLogFileSymlink
protected boolean generateLogFileSymlink(java.lang.String logFile, java.lang.String symlink)
This method generates a symlink to the actual log file written in the local temp directory. The symlink is placed in the dag directory.- Parameters:
logFile
- the full path to the log file.symlink
- the full path to the symlink.- Returns:
- boolean indicating if creation of symlink was successful or not
-
getDAGMANOutFilename
protected java.lang.String getDAGMANOutFilename(ADag dag)
Returns the basename of the file, that contains the output of the dagman while running the dag generated for the workflow. The basename of the .out file is dependant on whether the basename prefix has been specified at runtime or not by command line options.- Parameters:
dag
- the DAG containing the concrete workflow- Returns:
- the name of the dagfile.
-
concreteDagEmpty
protected void concreteDagEmpty(java.lang.String filename, ADag dag) throws CodeGeneratorException
A callout method that dictates what needs to be done in case the concrete plan that is generated is empty. It just logs a message saying the plan is empty.- Parameters:
filename
- Filename of the dag to be written of type String.dag
- the concrete dag that is empty.- Throws:
CodeGeneratorException
- in case of any error occuring code generation.
-
handleCondorVarForJob
protected void handleCondorVarForJob(Job job) throws CodeGeneratorException
It updates/adds the condor variables that are got through the Dax with the values specified in the properties file, pool config file or adds some variables internally. In case of clashes of Condor variables from various sources the following order is followed,property file, pool config file and then dax.- Parameters:
job
- The Job object containing the information about the job.- Throws:
CodeGeneratorException
-
populatePeriodicReleaseAndRemoveInJob
public void populatePeriodicReleaseAndRemoveInJob(Job job)
Populates the periodic release and remove values in the job. If an integer value is specified it is used to construct the default expression, else the value specified in the profiles is used as is. The default expression for periodic_release and periodic_remove isperiodic_release = False periodic_remove = (JobStatus == 5) && ((CurrentTime - EnteredCurrentStatus) > 14400)
where releasevalue is value of condor profile periodic_release and removevalue is value of condor profile periodic_remove- Parameters:
job
- the job object.
-
isInteger
protected boolean isInteger(java.lang.String value)
Returns a boolean indicating whether the value represented is an integer or not.- Parameters:
value
- the String passed- Returns:
- true if an int else false
-
applyStyle
protected void applyStyle(Job job, java.io.PrintWriter writer) throws CodeGeneratorException
Applies a submit file style to the job, according to the fact whether the job has to be submitted directly to condor or to a remote jobmanager via CondorG and GRAM. If no style is associated with the job, then for the job running on local site, condor style is applied. For a job running on non local sites, globus style is applied if none is associated with the job.- Parameters:
job
- the job on which the style needs to be applied.writer
- the PrintWriter stream to the submit file for the job.- Throws:
CodeGeneratorException
- in case of any error occuring code generation.
-
handleEnvVarForJob
protected void handleEnvVarForJob(Job sinfo)
It updates/adds the environment variables that are got through the Dax with the values specified in the properties file, pool config file or adds some variables internally. In case of clashes of environment variables from various sources the following order is followed,property file, transformation catalog, pool config file and then dax. At present values are not picked from the properties file.- Parameters:
job
- The Job object containing the information about the job.
-
handleGlobusRSLForJob
protected void handleGlobusRSLForJob(Job sinfo)
It updates/adds the the Globus RSL parameters got through the dax that are in Job object. In addition inserts the additional rsl attributes that can be specified in the properties file or the pool config files in the profiles tags. In case of clashes of RSL attributes from various sources the following order is followed,property file, pool config file and then dax.- Parameters:
job
- The Job object containing the information about the job.
-
getJobPriority
protected int getJobPriority(Job job, int depth)
Computes the priority for a job based on job type and depth in the workflow- Parameters:
job
- the job whose priority needs to be computeddepth
- the depth in the workflow- Returns:
-
gridstart
private java.lang.String gridstart(java.io.PrintWriter writer, Job job, boolean isGlobusJob) throws CodeGeneratorException
This function creates the stdio handling with and without gridstart. Please note that gridstart will become the default by end 2003, and no gridstart support will be phased out.- Parameters:
writer
- is an open stream for the Condor submit file.job
- is the job information structure.isGlobusJob
- istrue
, if the job generated a lineuniverse = globus
, and thus runs remotely. Set tofalse
, if the job runs on the submit host in any way.- Returns:
- A possibly empty string which contains things that need to be added to the "globusrsl" clause. The return value is only of interest for isGlobusJob==true calls.
- Throws:
CodeGeneratorException
- in case of any error occuring code generation.
-
-