Class ReplicaCatalogBridge


  • public class ReplicaCatalogBridge
    extends Engine
    This coordinates the look up to the Replica Location Service, to determine the logical to physical mappings.
    Version:
    $Revision$
    Author:
    Karan Vahi, Gaurang Mehta
    • Field Detail

      • DEFAULT_REGISTRATION_CATEGORY_KEY

        public static final java.lang.String DEFAULT_REGISTRATION_CATEGORY_KEY
        Default category for registration jobs
        See Also:
        Constant Field Values
      • RC_TRANSFORMATION_NS

        public static final java.lang.String RC_TRANSFORMATION_NS
        The transformation namespace for the regostration jobs.
        See Also:
        Constant Field Values
      • RC_TRANSFORMATION_NAME

        public static final java.lang.String RC_TRANSFORMATION_NAME
        The logical name of the transformation used.
        See Also:
        Constant Field Values
      • RC_TRANSFORMATION_VERSION

        public static final java.lang.String RC_TRANSFORMATION_VERSION
        The logical name of the transformation used.
      • RC_DERIVATION_NS

        public static final java.lang.String RC_DERIVATION_NS
        The derivation namespace for the transfer jobs.
        See Also:
        Constant Field Values
      • RC_DERIVATION_NAME

        public static final java.lang.String RC_DERIVATION_NAME
        The derivation name for the transfer jobs.
        See Also:
        Constant Field Values
      • RC_DERIVATION_VERSION

        public static final java.lang.String RC_DERIVATION_VERSION
        The version number for the derivations for registration jobs.
        See Also:
        Constant Field Values
      • CACHE_REPLICA_CATALOG_IMPLEMENTER

        public static final java.lang.String CACHE_REPLICA_CATALOG_IMPLEMENTER
        The name of the Replica Catalog Implementer that serves as the source for cache files.
        See Also:
        Constant Field Values
      • CACHE_REPLICA_CATALOG_KEY

        public static final java.lang.String CACHE_REPLICA_CATALOG_KEY
        The name of the source key for Replica Catalog Implementer that serves as cache
        See Also:
        Constant Field Values
      • CACHE_READ_ONLY_KEY

        public static final java.lang.String CACHE_READ_ONLY_KEY
        The name of the key that disables writing back to the cache file. Designates a static file. i.e. read only
        See Also:
        Constant Field Values
      • DIRECTORY_REPLICA_CATALOG_IMPLEMENTER

        public static final java.lang.String DIRECTORY_REPLICA_CATALOG_IMPLEMENTER
        The name of the Replica Catalog Implementer that serves as the source for cache files.
        See Also:
        Constant Field Values
      • DIRECTORY_REPLICA_CATALOG_KEY

        public static final java.lang.String DIRECTORY_REPLICA_CATALOG_KEY
        The name of the source key for Replica Catalog Implementer that serves as cache
        See Also:
        Constant Field Values
      • REPLICA_CATALOG_URL_KEY

        public static final java.lang.String REPLICA_CATALOG_URL_KEY
        The name of the URL key for the replica catalog impelementer to be picked up.
        See Also:
        Constant Field Values
      • mReplicaCatalog

        private ReplicaCatalog mReplicaCatalog
        The handle to the main Replica Catalog.
      • mSearchFiles

        protected java.util.Set mSearchFiles
        The Vector of String objects containing the logical filenames of the files whose locations are to be searched in the Replica Catalog.
      • mRCDown

        private boolean mRCDown
        A boolean variable to desingnate whether the RLI queried was down or not. By default it is up, unless it is set to true explicitly.
      • mReplicaStore

        private ReplicaStore mReplicaStore
        The replica store in which we store all the results that are queried from the main replica catalog.
      • mCacheStore

        private ReplicaStore mCacheStore
        The replica store in which we store all the results that are queried from the cache replica catalogs.
      • mDirectoryReplicaStore

        private ReplicaStore mDirectoryReplicaStore
        The replica store where we store all the results that are queried from the input directory specified at the command line.
      • mDAXReplicaStore

        private ReplicaStore mDAXReplicaStore
        The DAX Replica Store.
      • mInheritedReplicaStore

        private ReplicaStore mInheritedReplicaStore
        The inherited Replica Store
      • mTreatCacheAsRC

        private boolean mTreatCacheAsRC
        A boolean indicating whether the cache file needs to be treated as a replica catalog or not.
      • mDefaultTCRCCreated

        private boolean mDefaultTCRCCreated
        A boolean indicating whether the attempt to create a default tc entry has happened or not.
      • mDag

        private ADag mDag
        The DAG being worked upon.
    • Constructor Detail

      • ReplicaCatalogBridge

        public ReplicaCatalogBridge​(ADag dag,
                                    PegasusBag bag)
        The overloaded constructor.
        Parameters:
        dag - the workflow that is being worked on.
        bag - of initialization objects.
    • Method Detail

      • initialize

        public void initialize​(ADag dag,
                               PegasusBag bag)
        Intialises the refiner.
        Parameters:
        dag - the workflow that is being worked on.
        bag - the bag of Pegasus initialization objects
      • initialize

        public void initialize​(ADag dag,
                               PegasusProperties properties,
                               PlannerOptions options)
        Intialises the refiner.
        Parameters:
        dag - the workflow that is being worked on.
        properties - the properties passed to the planner.
        options - the options passed to the planner at runtime.
      • closeConnection

        public void closeConnection()
        To close the connection to replica services. This must be defined in the case where one has not done a singleton implementation. In other cases just do an empty implementation of this method.
      • finalize

        public void finalize()
        Closes the connection to the rli.
        Overrides:
        finalize in class java.lang.Object
      • getFilesInReplica

        public java.util.Set getFilesInReplica()
        This returns the files for which mappings exist in the Replica Catalog. This should return a subset of the files which are specified in the mSearchFiles, while getting an instance to this.
        Returns:
        a Set of logical file names as String objects, for which logical to physical mapping exists.
        See Also:
        mSearchFiles
      • getFileLocs

        public ReplicaLocation getFileLocs​(java.lang.String lfn)
        Returns all the locations as returned from the Replica Lookup Mechanism.
        Parameters:
        lfn - The name of the logical file whose PFN mappings are required.
        Returns:
        ReplicaLocation containing all the locations for that LFN
        See Also:
        org.griphyn.cPlanner.classes.ReplicaLocation
      • getDefaultRegistrationMaxJobsPropertyKey

        public java.lang.String getDefaultRegistrationMaxJobsPropertyKey()
        Returns the property key that can be used to set the max jobs for the default category associated with the registration jobs.
        Returns:
        the property key
      • makeRCRegNode

        public Job makeRCRegNode​(java.lang.String regJobName,
                                 Job job,
                                 java.util.Collection files)
        It constructs the Job object for the registration node, which registers the materialized files on the output pool in the RLS. Note that the relations corresponding to this node should already have been added to the concerned DagInfo object.
        Parameters:
        regJobName - The name of the job which registers the files in the Replica Location Service.
        job - The job whose output files are to be registered in the Replica Location Service.
        files - Collection of FileTransfer objects containing the information about source and destination URLs. The destination URLs would be our PFNs.
        Returns:
        Job corresponding to the new registration node.
      • defaultTCRCEntry

        private TransformationCatalogEntry defaultTCRCEntry()
        Returns a default TC entry to be used in case entry is not found in the transformation catalog.
        Returns:
        the default entry.
      • getClassPath

        private Profile getClassPath​(java.lang.String home)
        Returns the classpath for the default rc-client entry.
        Parameters:
        home - the home directory where we need to check for lib directory.
        Returns:
        the classpath in an environment profile.
      • generateRepJobArgumentString

        private java.lang.String generateRepJobArgumentString​(SiteCatalogEntry site,
                                                              java.lang.String regJob,
                                                              java.util.Collection files)
        Generates the argument string to be given to the replica registration job. At present by default it would be picking up the file containing the mappings.
        Parameters:
        site - the SiteCatalogEntry object
        regJob - The name of the registration job.
        files - Collection of FileTransfer objects containing the information about source and destURLs. The destination URLs would be our PFNs.
        Returns:
        the argument string.
      • getCommandLineProperties

        protected java.lang.String getCommandLineProperties​(PegasusProperties properties)
        Returns the properties that need to be passed to the the rc-client invocation on the command line . It is of the form "-Dprop1=value1 -Dprop2=value2 .."
        Parameters:
        properties - the properties object
        Returns:
        the properties list, else empty string.
      • appendProperty

        protected void appendProperty​(java.lang.StringBuffer sb,
                                      java.lang.String key,
                                      java.lang.String value)
        Appends a property to the StringBuffer, in the java command line format.
        Parameters:
        sb - the StringBuffer to append the property to.
        key - the property.
        value - the property value.
      • generateMappingsFile

        private java.lang.String generateMappingsFile​(java.lang.String regJob,
                                                      java.util.Collection files)
        Generates the registration mappings in a text file that is generated in the dax directory (the directory where all the condor submit files are generated). The pool value for the mapping is the output pool specified by the user when running Pegasus. The name of the file is regJob+.in
        Parameters:
        regJob - The name of the registration job.
        files - Collection of FileTransferobjects containing the information about source and destURLs. The destination URLs would be our PFNs.
        Returns:
        String corresponding to the path of the the file containig the mappings in the appropriate format.
      • ftToRC

        private java.lang.String ftToRC​(FileTransfer ft)
        Converts a FileTransfer to a RC compatible string representation.
        Parameters:
        ft - the FileTransfer object
        Returns:
        the RC version.
      • retrieveFromCache

        private ReplicaLocation retrieveFromCache​(java.lang.String lfn)
        Retrieves a location from the cache table, that contains the contents of the cache files specified at runtime.
        Parameters:
        lfn - the logical name of the file.
        Returns:
        ReplicaLocation object corresponding to the entry if found, else null.
      • loadInheritedReplicaStore

        private void loadInheritedReplicaStore​(java.util.Set files)
        Ends up loading the inherited replica files.
        Parameters:
        files - set of paths to the inherited replica files.
      • loadCacheFiles

        private void loadCacheFiles​(java.util.Set cacheFiles)
        Ends up loading the cache files so as to enable the lookup for the transient files created by the parent jobs.
        Parameters:
        cacheFiles - set of paths to the cache files.
      • getReplicaStoreFromFiles

        private ReplicaStore getReplicaStoreFromFiles​(java.util.Set files)
        Ends up loading a Replica Store from replica catalog files
        Parameters:
        files - set of paths to the cache files.
      • getReplicaStoreFromDirectory

        private ReplicaStore getReplicaStoreFromDirectory​(java.lang.String directory)
        Loads the mappings from the input directory
        Parameters:
        directory - the directory to load from
      • getPathToLocalProxy

        private java.lang.String getPathToLocalProxy()
        Returns path to the local proxy
        Returns:
        path to the local proxy