Class CondorGenerator

  • All Implemented Interfaces:
    CodeGenerator
    Direct Known Subclasses:
    HashedFile

    public class CondorGenerator
    extends Abstract
    This class generates the condor submit files for the DAG which has to be submitted to the Condor DagMan.
    Version:
    $Revision$
    Author:
    Gaurang Mehta, Karan Vahi
    • Constructor Summary

      Constructors 
      Constructor Description
      CondorGenerator()
      The default constructor.
    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      protected void applyStyle​(Job job, java.io.PrintWriter writer)
      Applies a submit file style to the job, according to the fact whether the job has to be submitted directly to condor or to a remote jobmanager via CondorG and GRAM.
      protected void concreteDagEmpty​(java.lang.String filename, ADag dag)
      A callout method that dictates what needs to be done in case the concrete plan that is generated is empty.
      protected Job constructDAGJob​(java.lang.String name, java.lang.String directory, java.lang.String dagBasename)
      Constructs a job that plans and submits the partitioned workflow, referred to by a Partition.
      private java.util.Map constructMap​(java.lang.String propValue)
      Constructs a map with the numbers/values to be passed in the RSL handle for certain pools.
      private TransformationCatalogEntry constructTCEntryFromEnvironment()
      Returns a tranformation catalog entry object constructed from the environment An entry is constructed if either of the following environment variables are defined 1) CONDOR_HOME 2) CONDOR_LOCATION CONDOR_HOME takes precedence over CONDOR_LOCATION
      private TransformationCatalogEntry constructTCEntryFromEnvProfiles​(ENV env)
      Returns a tranformation catalog entry object constructed from the environment An entry is constructed if either of the following environment variables are defined 1) CONDOR_HOME 2) CONDOR_LOCATION CONDOR_HOME takes precedence over CONDOR_LOCATION
      private TransformationCatalogEntry defaultTCEntry​(java.lang.String site)
      Returns a default TC entry to be used in case entry is not found in the transformation catalog.
      java.util.Collection<java.io.File> generateCode​(ADag dag)
      Generates the code for the concrete workflow in Condor DAGMAN and CondorG input format.
      void generateCode​(ADag dag, Job job)
      Generates the code (condor submit file) for a single job.
      protected boolean generateLogFileSymlink​(java.lang.String logFile, java.lang.String symlink)
      This method generates a symlink to the actual log file written in the local temp directory.
      java.util.Map<java.lang.String,​java.lang.String> getAdditionalBraindumpEntries​(ADag workflow)
      Returns a Map containing additional braindump entries that are specific to a Code Generator.
      protected java.lang.String getBasename​(java.lang.String prefix, java.lang.String suffix)
      A covenience method to construct the basename.
      protected java.lang.String getCategoryDAGManKnobs​(PegasusProperties properties)
      Write out the DAGMan knobs for each category the user mentions in the properties.
      protected java.lang.String getCondorLogInSubmitDirectory()
      Returns the path to the condor log file in the submit directory.
      protected java.lang.String getCondorLogInSubmitDirectory​(ADag dag)
      Returns the path to the condor log file in the submit directory.
      protected java.lang.String getCondorLogInTmpDirectory()
      Returns the name of Condor log file in a tmp directory that is created if generation of symlink for condor logs is turned on.
      protected java.lang.String getDAGMANOutFilename​(ADag dag)
      Returns the basename of the file, that contains the output of the dagman while running the dag generated for the workflow.
      protected int getJobPriority​(Job job, int depth)
      Computes the priority for a job based on job type and depth in the workflow
      private java.lang.String gridstart​(java.io.PrintWriter writer, Job job, boolean isGlobusJob)
      This function creates the stdio handling with and without gridstart.
      protected void handleCondorVarForJob​(Job job)
      It updates/adds the condor variables that are got through the Dax with the values specified in the properties file, pool config file or adds some variables internally.
      protected void handleEnvVarForJob​(Job sinfo)
      It updates/adds the environment variables that are got through the Dax with the values specified in the properties file, pool config file or adds some variables internally.
      protected void handleGlobusRSLForJob​(Job sinfo)
      It updates/adds the the Globus RSL parameters got through the dax that are in Job object.
      void initialize​(PegasusBag bag)
      Initializes the Code Generator implementation.
      protected java.io.File initializeDagFileWriter​(java.lang.String filename, ADag workflow)
      Initializes the file handler to the dag file and writes the header to it.
      protected boolean isInteger​(java.lang.String value)
      Returns a boolean indicating whether the value represented is an integer or not.
      void populatePeriodicReleaseAndRemoveInJob​(Job job)
      Populates the periodic release and remove values in the job.
      protected void printDagString​(java.lang.String str)
      Writes a string to the dag file.
      void reset()
      Resets the Code Generator implementation.
      boolean startMonitoring()
      Starts monitoring of the workflow by invoking a workflow monitor daemon tailstatd.
      protected void writeDagFileTail​(ADag dag)
      It writes the relations making up the DAG in the dag file and and closes the file handle to it.
      protected void writeDOTFile​(java.lang.String filename, ADag dag)
      Writes out the DOT file in the submit directory.
      protected void writeJobMapFile​(java.lang.String filename, ADag dag)
      Writes out the job map file in the submit directory.
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Field Detail

      • DEFAULT_SUBDAG_CATEGORY_KEY

        public static final java.lang.String DEFAULT_SUBDAG_CATEGORY_KEY
        The default category for the sub dax jobs.
        See Also:
        Constant Field Values
      • mSeparator

        public static final java.lang.String mSeparator
        The nice separator, define once, use often.
        See Also:
        Constant Field Values
      • DEFAULT_PERIODIC_RELEASE_VALUE

        public static final java.lang.String DEFAULT_PERIODIC_RELEASE_VALUE
        Default value for the periodic_release for a job
        See Also:
        Constant Field Values
      • DEFAULT_PERIODIC_REMOVE_VALUE

        public static final java.lang.String DEFAULT_PERIODIC_REMOVE_VALUE
        Default value for the periodic_remove for a job
        See Also:
        Constant Field Values
      • CONDOR_DAGMAN_NAMESPACE

        public static final java.lang.String CONDOR_DAGMAN_NAMESPACE
        The namespace to use for condor dagman.
        See Also:
        Constant Field Values
      • CONDOR_DAGMAN_LOGICAL_NAME

        public static final java.lang.String CONDOR_DAGMAN_LOGICAL_NAME
        The logical name with which to query the transformation catalog for the condor_dagman executable, that ends up running the mini dag as one job.
        See Also:
        Constant Field Values
      • DAGMAN_PROPERTIES_PREFIX

        public static final java.lang.String DAGMAN_PROPERTIES_PREFIX
        The prefix for DAGMan specific properties
        See Also:
        Constant Field Values
      • DEFAULT_STAGE_IN_PRIORITY_KEY

        public static final int DEFAULT_STAGE_IN_PRIORITY_KEY
        The default priority key associated with the stagein jobs.
        See Also:
        Constant Field Values
      • DEFAULT_INTER_SITE_PRIORITY_KEY

        public static final int DEFAULT_INTER_SITE_PRIORITY_KEY
        The default priority key associated with the inter site transfer jobs.
        See Also:
        Constant Field Values
      • DEFAULT_CREATE_DIR_PRIORITY_KEY

        public static final int DEFAULT_CREATE_DIR_PRIORITY_KEY
        The default priority key associated with the create dir jobs.
        See Also:
        Constant Field Values
      • DEFAULT_CHMOD_PRIORITY_KEY

        public static final int DEFAULT_CHMOD_PRIORITY_KEY
        The default priority key associated with chmod jobs.
        See Also:
        Constant Field Values
      • DEFAULT_STAGE_OUT_PRIORITY_KEY

        public static final int DEFAULT_STAGE_OUT_PRIORITY_KEY
        The default priority key associated with the stage out jobs.
        See Also:
        Constant Field Values
      • DEFAULT_REPLICA_REG_PRIORITY_KEY

        public static final int DEFAULT_REPLICA_REG_PRIORITY_KEY
        The default priority key associated with the replica registration jobs.
        See Also:
        Constant Field Values
      • DEFAULT_CLEANUP_PRIORITY_KEY

        public static final int DEFAULT_CLEANUP_PRIORITY_KEY
        The default priority key associated with the cleanup jobs.
        See Also:
        Constant Field Values
      • mSiteStore

        private SiteStore mSiteStore
        The handle to the site catalog store.
      • mPoolClass

        protected java.lang.String mPoolClass
        Specifies the implementing class for the pool interface. Contains the name of the class that implements the pool interface the user has asked at runtime.
      • mDagWriter

        protected java.io.PrintWriter mDagWriter
        The file handle to the .dag file. A part of the dag file is printed as we write the submit files, to insert the appropriate postscripts for handling exit codes.
      • mTempLogFile

        protected java.lang.String mTempLogFile
        The name of the log file in the /tmp directory
      • mDone

        protected boolean mDone
        A boolean indicating whether the files have been generated or not.
      • mConcreteWorkflow

        protected ADag mConcreteWorkflow
        The workflow for which the code has to be generated.
      • mStyleFactory

        protected CondorStyleFactory mStyleFactory
        Handle to the Style factory, that is used for this workflow.
      • mGridStartFactory

        protected GridStartFactory mGridStartFactory
        The handle to the GridStart Factory.
      • mInitializeGridStart

        protected boolean mInitializeGridStart
        A boolean indicating whether grid start has been initialized or not.
      • mCondorVersion

        private long mCondorVersion
        The long value of condor version.
      • mAssignDefaultJobPriorities

        private boolean mAssignDefaultJobPriorities
        Boolean indicating whether to assign job priorities or not.
    • Constructor Detail

      • CondorGenerator

        public CondorGenerator()
        The default constructor.
    • Method Detail

      • generateCode

        public java.util.Collection<java.io.File> generateCode​(ADag dag)
                                                        throws CodeGeneratorException
        Generates the code for the concrete workflow in Condor DAGMAN and CondorG input format. Returns only the File object for the DAG file that is written out.
        Parameters:
        dag - the concrete workflow.
        Returns:
        the Collection of File objects for the files written out.
        Throws:
        CodeGeneratorException - in case of any error occuring code generation.
      • generateCode

        public void generateCode​(ADag dag,
                                 Job job)
                          throws CodeGeneratorException
        Generates the code (condor submit file) for a single job.
        Parameters:
        dag - the dag of which the job is a part of.
        job - the Job object holding the information about that particular job.
        Throws:
        CodeGeneratorException - in case of any error occuring code generation.
      • startMonitoring

        public boolean startMonitoring()
                                throws org.griphyn.vdl.euryale.VTorInUseException
        Starts monitoring of the workflow by invoking a workflow monitor daemon tailstatd. The tailstatd is picked up from the default path of $PEGASUS_HOME/bin/tailstatd.
        Specified by:
        startMonitoring in interface CodeGenerator
        Overrides:
        startMonitoring in class Abstract
        Returns:
        boolean indicating whether could successfully start the monitor daemon or not.
        Throws:
        org.griphyn.vdl.euryale.VTorInUseException - in case the method is called before the submit files have been generated.
      • constructMap

        private java.util.Map constructMap​(java.lang.String propValue)
        Constructs a map with the numbers/values to be passed in the RSL handle for certain pools. The user ends up specifying these through the properties file. The value of the property is of the form poolname1=value,poolname2=value....
        Parameters:
        propValue - the value of the property got from the properties file.
        Returns:
        Map
      • constructDAGJob

        protected Job constructDAGJob​(java.lang.String name,
                                      java.lang.String directory,
                                      java.lang.String dagBasename)
        Constructs a job that plans and submits the partitioned workflow, referred to by a Partition. The main job itself is a condor dagman job that submits the concrete workflow. The concrete workflow is generated by running the planner in the prescript for the job.
        Parameters:
        name - the name to be assigned to the job.
        directory - the submit directory where the submit files for the partition should reside. this is where the dag file is created
        dagBasename - the basename of the dag file created.
        Returns:
        the constructed DAG job.
      • defaultTCEntry

        private TransformationCatalogEntry defaultTCEntry​(java.lang.String site)
        Returns a default TC entry to be used in case entry is not found in the transformation catalog.
        Parameters:
        site - the site for which the default entry is required.
        Returns:
        the default entry.
      • constructTCEntryFromEnvironment

        private TransformationCatalogEntry constructTCEntryFromEnvironment()
        Returns a tranformation catalog entry object constructed from the environment An entry is constructed if either of the following environment variables are defined 1) CONDOR_HOME 2) CONDOR_LOCATION CONDOR_HOME takes precedence over CONDOR_LOCATION
        Returns:
        the constructed entry else null.
      • constructTCEntryFromEnvProfiles

        private TransformationCatalogEntry constructTCEntryFromEnvProfiles​(ENV env)
        Returns a tranformation catalog entry object constructed from the environment An entry is constructed if either of the following environment variables are defined 1) CONDOR_HOME 2) CONDOR_LOCATION CONDOR_HOME takes precedence over CONDOR_LOCATION
        Parameters:
        env - the environment profiles.
        Returns:
        the entry constructed else null if environment variables not defined.
      • getBasename

        protected java.lang.String getBasename​(java.lang.String prefix,
                                               java.lang.String suffix)
        A covenience method to construct the basename.
        Parameters:
        prefix - the first half of basename
        suffix - the latter half of basename
        Returns:
        basename
      • initializeDagFileWriter

        protected java.io.File initializeDagFileWriter​(java.lang.String filename,
                                                       ADag workflow)
                                                throws CodeGeneratorException
        Initializes the file handler to the dag file and writes the header to it.
        Parameters:
        filename - basename of dag file to be written.
        dag - the workflow
        Returns:
        the File object for the DAG file.
        Throws:
        CodeGeneratorException - in case of any error occuring code generation.
      • getCategoryDAGManKnobs

        protected java.lang.String getCategoryDAGManKnobs​(PegasusProperties properties)
        Write out the DAGMan knobs for each category the user mentions in the properties.
        Parameters:
        properties - the pegasus properties
        Returns:
        the String
      • writeDOTFile

        protected void writeDOTFile​(java.lang.String filename,
                                    ADag dag)
                             throws CodeGeneratorException
        Writes out the DOT file in the submit directory.
        Parameters:
        filename - basename of dot file to be written .
        dag - the ADag object.
        Throws:
        CodeGeneratorException - in case of any error occuring code generation.
      • writeJobMapFile

        protected void writeJobMapFile​(java.lang.String filename,
                                       ADag dag)
                                throws CodeGeneratorException
        Writes out the job map file in the submit directory.
        Parameters:
        filename - basename of dot file to be written .
        dag - the ADag object.
        Throws:
        CodeGeneratorException - in case of any error occuring code generation.
      • writeDagFileTail

        protected void writeDagFileTail​(ADag dag)
                                 throws CodeGeneratorException
        It writes the relations making up the DAG in the dag file and and closes the file handle to it.
        Parameters:
        dag - the executable workflow
        Throws:
        CodeGeneratorException
      • printDagString

        protected void printDagString​(java.lang.String str)
                               throws CodeGeneratorException
        Writes a string to the dag file. When calling this function the file handle to file is already initialized.
        Parameters:
        str - The String to be printed to the dag file.
        Throws:
        CodeGeneratorException
      • getCondorLogInTmpDirectory

        protected java.lang.String getCondorLogInTmpDirectory()
        Returns the name of Condor log file in a tmp directory that is created if generation of symlink for condor logs is turned on.
        Returns:
        the name of the log file.
      • getCondorLogInSubmitDirectory

        protected java.lang.String getCondorLogInSubmitDirectory()
        Returns the path to the condor log file in the submit directory. It can be a symlink.
        Parameters:
        dag - the concrete workflow.
        Returns:
        the path to condor log file in the submit directory.
      • getCondorLogInSubmitDirectory

        protected java.lang.String getCondorLogInSubmitDirectory​(ADag dag)
        Returns the path to the condor log file in the submit directory. It can be a symlink.
        Parameters:
        dag - the concrete workflow.
        Returns:
        the path to condor log file in the submit directory.
      • getAdditionalBraindumpEntries

        public java.util.Map<java.lang.String,​java.lang.String> getAdditionalBraindumpEntries​(ADag workflow)
        Returns a Map containing additional braindump entries that are specific to a Code Generator.
        Specified by:
        getAdditionalBraindumpEntries in class Abstract
        Parameters:
        workflow - the executable workflow
        Returns:
        Map containing entries for dag and condor_log
      • generateLogFileSymlink

        protected boolean generateLogFileSymlink​(java.lang.String logFile,
                                                 java.lang.String symlink)
        This method generates a symlink to the actual log file written in the local temp directory. The symlink is placed in the dag directory.
        Parameters:
        logFile - the full path to the log file.
        symlink - the full path to the symlink.
        Returns:
        boolean indicating if creation of symlink was successful or not
      • getDAGMANOutFilename

        protected java.lang.String getDAGMANOutFilename​(ADag dag)
        Returns the basename of the file, that contains the output of the dagman while running the dag generated for the workflow. The basename of the .out file is dependant on whether the basename prefix has been specified at runtime or not by command line options.
        Parameters:
        dag - the DAG containing the concrete workflow
        Returns:
        the name of the dagfile.
      • concreteDagEmpty

        protected void concreteDagEmpty​(java.lang.String filename,
                                        ADag dag)
                                 throws CodeGeneratorException
        A callout method that dictates what needs to be done in case the concrete plan that is generated is empty. It just logs a message saying the plan is empty.
        Parameters:
        filename - Filename of the dag to be written of type String.
        dag - the concrete dag that is empty.
        Throws:
        CodeGeneratorException - in case of any error occuring code generation.
      • handleCondorVarForJob

        protected void handleCondorVarForJob​(Job job)
                                      throws CodeGeneratorException
        It updates/adds the condor variables that are got through the Dax with the values specified in the properties file, pool config file or adds some variables internally. In case of clashes of Condor variables from various sources the following order is followed,property file, pool config file and then dax.
        Parameters:
        job - The Job object containing the information about the job.
        Throws:
        CodeGeneratorException
      • populatePeriodicReleaseAndRemoveInJob

        public void populatePeriodicReleaseAndRemoveInJob​(Job job)
        Populates the periodic release and remove values in the job. If an integer value is specified it is used to construct the default expression, else the value specified in the profiles is used as is. The default expression for periodic_release and periodic_remove is
          periodic_release = False
          periodic_remove = (JobStatus == 5) && ((CurrentTime - EnteredCurrentStatus) > 14400)
         
        where releasevalue is value of condor profile periodic_release and removevalue is value of condor profile periodic_remove
        Parameters:
        job - the job object.
      • isInteger

        protected boolean isInteger​(java.lang.String value)
        Returns a boolean indicating whether the value represented is an integer or not.
        Parameters:
        value - the String passed
        Returns:
        true if an int else false
      • applyStyle

        protected void applyStyle​(Job job,
                                  java.io.PrintWriter writer)
                           throws CodeGeneratorException
        Applies a submit file style to the job, according to the fact whether the job has to be submitted directly to condor or to a remote jobmanager via CondorG and GRAM. If no style is associated with the job, then for the job running on local site, condor style is applied. For a job running on non local sites, globus style is applied if none is associated with the job.
        Parameters:
        job - the job on which the style needs to be applied.
        writer - the PrintWriter stream to the submit file for the job.
        Throws:
        CodeGeneratorException - in case of any error occuring code generation.
      • handleEnvVarForJob

        protected void handleEnvVarForJob​(Job sinfo)
        It updates/adds the environment variables that are got through the Dax with the values specified in the properties file, pool config file or adds some variables internally. In case of clashes of environment variables from various sources the following order is followed,property file, transformation catalog, pool config file and then dax. At present values are not picked from the properties file.
        Parameters:
        job - The Job object containing the information about the job.
      • handleGlobusRSLForJob

        protected void handleGlobusRSLForJob​(Job sinfo)
        It updates/adds the the Globus RSL parameters got through the dax that are in Job object. In addition inserts the additional rsl attributes that can be specified in the properties file or the pool config files in the profiles tags. In case of clashes of RSL attributes from various sources the following order is followed,property file, pool config file and then dax.
        Parameters:
        job - The Job object containing the information about the job.
      • getJobPriority

        protected int getJobPriority​(Job job,
                                     int depth)
        Computes the priority for a job based on job type and depth in the workflow
        Parameters:
        job - the job whose priority needs to be computed
        depth - the depth in the workflow
        Returns:
      • gridstart

        private java.lang.String gridstart​(java.io.PrintWriter writer,
                                           Job job,
                                           boolean isGlobusJob)
                                    throws CodeGeneratorException
        This function creates the stdio handling with and without gridstart. Please note that gridstart will become the default by end 2003, and no gridstart support will be phased out.
        Parameters:
        writer - is an open stream for the Condor submit file.
        job - is the job information structure.
        isGlobusJob - is true, if the job generated a line universe = globus, and thus runs remotely. Set to false, if the job runs on the submit host in any way.
        Returns:
        A possibly empty string which contains things that need to be added to the "globusrsl" clause. The return value is only of interest for isGlobusJob==true calls.
        Throws:
        CodeGeneratorException - in case of any error occuring code generation.