Class Horizontal
- java.lang.Object
-
- edu.isi.pegasus.planner.partitioner.Partitioner
-
- edu.isi.pegasus.planner.partitioner.BFS
-
- edu.isi.pegasus.planner.partitioner.Horizontal
-
public class Horizontal extends BFS
Horizontal based partitioning scheme, that allows the user to configure the number of partitions per transformation name per level. To set the size of the partition per transformation, the following properties need to be setpegasus.partitioner.horizontal.collapse.[txName] pegasus.partitioner.horizontal.bundle.[txName]
The bundle value designates the number of partitions per transformation per level. The collapse values designates the number of nodes in a partitioning referring to a particular transformation. If both are specified, then bundle value takes precedence.- Version:
- $Revision$
- Author:
- Karan Vahi
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description private static class
Horizontal.GraphNodeComparator
A GraphNode comparator, that allows us to compare nodes according to the transformation logical names.
-
Field Summary
Fields Modifier and Type Field Description static int
DEFAULT_COLLAPSE_FACTOR
The default collapse factor for collapsing jobs with same logical name scheduled onto the same execution pool.static java.lang.String
DESCRIPTION
A short description about the partitioner.private int
mIDCounter
The global counter that is used to assign ID's to the partitions.private Horizontal.GraphNodeComparator
mNodeComparator
A static instance of GraphNode comparator.private java.util.Map
mPartitionMap
A map indexed by the partition ID.-
Fields inherited from class edu.isi.pegasus.planner.partitioner.Partitioner
mGraph, mLogger, mProps, mRoot, PACKAGE_NAME, VERSION
-
-
Constructor Summary
Constructors Constructor Description Horizontal(GraphNode root, java.util.Map graph, PegasusProperties properties)
The overloaded constructor.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description protected void
constructLevelRelations(Callback c, int parent, int child)
Calls out to the callback with appropriate relations between the partitions constructed for the levels.protected void
constructPartitions(Callback c, java.util.List nodes, int level)
Given a list of jobs, constructs (one or more) partitions out of it.protected void
constructPartitions(Callback c, java.util.List nodes, int level, java.lang.String name)
Given a list of jobs, constructs (one or more) partitions out of it.protected Partition
createPartition(java.util.List nodes)
Creates a partition out of a list of nodes.java.lang.String
description()
Returns a textual description of the partitioner implementation.protected void
done(Callback c)
Indicates that we are done with the traversal of the graph.protected int[]
getCollapseFactor(java.lang.String txName, int size)
Returns the collapse factor, that is used to determine the number of nodes going in a partition.private java.lang.String
getPartitionID(int id)
Constructs the id for the partition.private int
idCounter()
Returns the current value of the ID counter.private void
incrementIDCounter()
Increments the ID counter by 1.private java.util.Comparator
nodeComparator()
Singleton access to the job comparator.-
Methods inherited from class edu.isi.pegasus.planner.partitioner.BFS
determinePartitions
-
-
-
-
Field Detail
-
DESCRIPTION
public static final java.lang.String DESCRIPTION
A short description about the partitioner.- See Also:
- Constant Field Values
-
DEFAULT_COLLAPSE_FACTOR
public static final int DEFAULT_COLLAPSE_FACTOR
The default collapse factor for collapsing jobs with same logical name scheduled onto the same execution pool.- See Also:
- Constant Field Values
-
mPartitionMap
private java.util.Map mPartitionMap
A map indexed by the partition ID. Each value is a partition object.
-
mNodeComparator
private Horizontal.GraphNodeComparator mNodeComparator
A static instance of GraphNode comparator.
-
mIDCounter
private int mIDCounter
The global counter that is used to assign ID's to the partitions.
-
-
Constructor Detail
-
Horizontal
public Horizontal(GraphNode root, java.util.Map graph, PegasusProperties properties)
The overloaded constructor.- Parameters:
root
- the dummy root node of the graph.graph
- the map containing all the nodes of the graph keyed by the logical id of the nodes.properties
- the properties passed to the planner.
-
-
Method Detail
-
nodeComparator
private java.util.Comparator nodeComparator()
Singleton access to the job comparator.- Returns:
- the job comparator.
-
description
public java.lang.String description()
Returns a textual description of the partitioner implementation.- Overrides:
description
in classBFS
- Returns:
- a short textual description
-
constructPartitions
protected void constructPartitions(Callback c, java.util.List nodes, int level)
Given a list of jobs, constructs (one or more) partitions out of it. Calls out to the partitioner callback, for each of the partitions constructed.- Overrides:
constructPartitions
in classBFS
- Parameters:
c
- the parititoner callbacknodes
- the list ofGraphNode
objects on a particular level.level
- the level as determined from the root of the workflow.
-
constructPartitions
protected void constructPartitions(Callback c, java.util.List nodes, int level, java.lang.String name)
Given a list of jobs, constructs (one or more) partitions out of it. Calls out to the partitioner callback, for each of the partitions constructed.- Parameters:
c
- the parititoner callbacknodes
- the list ofGraphNode
objects on a particular level, referring to the same transformation underneath.level
- the level as determined from the root of the workflow.name
- the transformation name
-
constructLevelRelations
protected void constructLevelRelations(Callback c, int parent, int child)
Calls out to the callback with appropriate relations between the partitions constructed for the levels. This is an empty implementation, as we do our own book-keeping in this partitioner to determine the relations between the partitions.- Overrides:
constructLevelRelations
in classBFS
- Parameters:
c
- the parititoner callbackparent
- the parent levelchild
- the child level.- See Also:
done( Callback )
-
done
protected void done(Callback c)
Indicates that we are done with the traversal of the graph. Determines the relations between the partitions constructed and calls out to the appropriate callback function
-
getCollapseFactor
protected int[] getCollapseFactor(java.lang.String txName, int size)
Returns the collapse factor, that is used to determine the number of nodes going in a partition. The collapse factor is determined by getting the collapse and the bundle values specified for the transformations in the properties file. There are two orthogonal notions of bundling and collapsing. In case the bundle key is specified, it ends up overriding the collapse key, and the bundle value is used to generate the collapse values. If both are not specified or null, then collapseFactor is set to size.- Parameters:
txName
- the logical transformation namesize
- the number of jobs that refer to the same logical transformation and are scheduled on the same execution pool.- Returns:
- int array of size 2 where :- int[0] is the the collapse factor (number of nodes in a partition) int[1] is the number of parititons for whom collapsing is int[0] + 1.
-
createPartition
protected Partition createPartition(java.util.List nodes)
Creates a partition out of a list of nodes. Also stores it in the internal partition map to track partitions later on. Associates the partition ID with each of the nodes making the partition also.- Parameters:
nodes
- the list ofGraphNodes
making the partition.- Returns:
- the partition out of those nodes.
-
incrementIDCounter
private void incrementIDCounter()
Increments the ID counter by 1.
-
idCounter
private int idCounter()
Returns the current value of the ID counter.
-
getPartitionID
private java.lang.String getPartitionID(int id)
Constructs the id for the partition.- Parameters:
id
- an integer ID.- Returns:
- the ID for the Partition.
-
-