java.lang.Runnable
, IndexStatisticsDaemon
public class IndexStatisticsDaemonImpl extends java.lang.Object implements IndexStatisticsDaemon, java.lang.Runnable
The need for updated statistics is currently determined when compiling a SELECT query. The unit of work is then scheduled with this daemon, and the work itself will be carried out in a separate thread. If the worker thread doesn't exist it is created, if it is idle the unit of work will be processed immediately, and if it is busy the unit of work has to wait in the queue.
The daemon code has a notion of a background task. If the update is run as a background task, it will try to affect other activity in the Derby database as little as possible. As far as possible, it will not set locks on the conglomerates it scans, and if it needs to take locks it will give up immediately if the locks cannot be obtained. In some cases it will also roll back to release locks already taken, ad then retry. Since we are accessing shared structures the background work may still interfere with the user activity in the database due to locking, but all such operations carried out by the daemon are of short duration.
The high level flow of an update to index statistics is:
List of possible improvements:
Implementation notes: List of potential cleanups before going into a release:
Modifier and Type | Class | Description |
---|---|---|
private static class |
IndexStatisticsDaemonImpl.KeyComparator |
Support class used to compare keys when scanning indexes.
|
Modifier and Type | Field | Description |
---|---|---|
private static boolean |
AS_BACKGROUND_TASK |
|
private static boolean |
AS_EXPLICIT_TASK |
|
private ContextManager |
ctxMgr |
The context manager for the worker thread.
|
private boolean |
daemonDisabled |
Tells if the daemon has been disabled.
|
private LanguageConnectionContext |
daemonLCC |
The language connection context for the worker thread.
|
private java.lang.String |
databaseName |
|
private Database |
db |
The database object for the database we are handling automatic index
statistics update for.
|
private java.lang.String |
dbOwner |
The name of the database owner.
|
private boolean |
doLog |
Tells if logging is enabled.
|
private boolean |
doTrace |
Tells if tracing is enabled.
|
private int |
errorsConsecutive |
Number of consecutive errors, used as a metric to decide if the damoen
should be automatically shut down.
|
private long |
errorsKnown |
|
private long |
errorsUnknown |
|
private HeaderPrintWriter |
logStream |
|
private static int |
MAX_QUEUE_LENGTH |
Maximum number of work units allowed in the queue.
|
private java.util.ArrayList<TableDescriptor> |
queue |
A list of tables that shall have their index statistics updated.
|
private java.lang.Thread |
runningThread |
The thread in which the index statistics refresh operation is being
executed, if any.
|
private long |
runTime |
The period of time (ms) for which the daemon has been doing active work.
|
boolean |
skipDisposableStats |
Tells if disposable stats should be generated, which will happen in
soft-upgrade mode or when the user asks us to revert to the old behavior.
|
private long |
timeOfCreation |
Specifies when the daemon was created.
|
private boolean |
traceToDerbyLog |
Tells if traces are written to the Derby log file.
|
private boolean |
traceToStdOut |
Tells if traces are written to standard out.
|
private java.lang.StringBuffer |
tsb |
|
private long |
wuProcessed |
|
private long |
wuRejectedDup |
|
private long |
wuRejectedFQ |
|
private long |
wuRejectedOther |
|
private long |
wuScheduled |
Constructor | Description |
---|---|
IndexStatisticsDaemonImpl(HeaderPrintWriter log,
boolean doLog,
java.lang.String traceLevel,
Database db,
java.lang.String userName,
java.lang.String databaseName) |
Creates a new daemon.
|
Modifier and Type | Method | Description |
---|---|---|
private boolean |
acceptWork(TableDescriptor td) |
Determines if the given work can be accepted.
|
private void |
appendRunStats(java.lang.StringBuffer sb) |
Appends runtime statistics to the given string buffer.
|
private static java.lang.String |
cardToStr(long[] cardinality) |
Produces a textual representation of the cardinality numbers.
|
private boolean |
dbAtLeast10_9(Database db) |
Tells if the database is 10.9 or newer.
|
private static java.lang.String |
extractIstatInfo(java.lang.Throwable t) |
Purely for debugging, to avoid printing too much info.
|
private static java.lang.String |
fmtScanTimes(long[][] timings) |
Format array of scan durations as a string.
|
private void |
generateStatistics(LanguageConnectionContext lcc,
TableDescriptor td) |
Generates index statistics for all indexes associated with the given
table descriptor.
|
private static ContextService |
getContextService() |
Privileged lookup of the ContextService.
|
private boolean |
handleExpectedErrors(TableDescriptor td,
StandardException se) |
Handles expected errors.
|
private boolean |
handleFatalErrors(ContextManager cm,
StandardException se) |
Handles fatal errors that will cause the daemon to be shut down.
|
private boolean |
handleUnexpectedErrors(TableDescriptor td,
StandardException se) |
Handles unexpected errors.
|
private void |
invalidateStatements(LanguageConnectionContext lcc,
TableDescriptor td,
boolean asBackgroundTask) |
Performs an invalidation action for the given table (the event being
statistics update).
|
private boolean |
isShuttingDown() |
Return true if we are being shutdown
|
private void |
log(boolean asBackgroundTask,
TableDescriptor td,
java.lang.String msg) |
|
private void |
log(boolean asBackgroundTask,
TableDescriptor td,
java.lang.Throwable t,
java.lang.String msg) |
Logs the information given.
|
private void |
logAlways(TableDescriptor td,
java.lang.Throwable t,
java.lang.String msg) |
Logs the information given.
|
private void |
processingLoop() |
Main processing loop which will compute statistics until the queue
of scheduled work units has been drained.
|
void |
run() |
Drives the statistics generation.
|
void |
runExplicitly(LanguageConnectionContext lcc,
TableDescriptor td,
ConglomerateDescriptor[] cds,
java.lang.String runContext) |
Runs the statistics update sequence explicitly as requested by the user.
|
void |
schedule(TableDescriptor td) |
Schedules an update of the index statistics for the specified table.
|
private void |
setHeapRowEstimate(TransactionController tc,
long tableId,
long rowEstimate) |
Sets the row estimate for the heap conglomerate.
|
private static void |
sleep(long ms) |
Puts the current thread to sleep for maximum
ms milliseconds. |
void |
stop() |
Stops the daemon.
|
private void |
trace(int indentLevel,
java.lang.String msg) |
|
private void |
updateIndexStatsMinion(LanguageConnectionContext lcc,
TableDescriptor td,
ConglomerateDescriptor[] cds,
boolean asBackgroundTask) |
Updates the index statistics for the given table and the specified
indexes.
|
private void |
writeUpdatedStats(LanguageConnectionContext lcc,
TableDescriptor td,
UUID index,
long numRows,
long[] cardinality,
boolean asBackgroundTask) |
Writes updated statistics for the specified index to the data dictionary.
|
private static final boolean AS_BACKGROUND_TASK
private static final boolean AS_EXPLICIT_TASK
private static final int MAX_QUEUE_LENGTH
private final HeaderPrintWriter logStream
private final boolean doLog
private final boolean doTrace
private final boolean traceToDerbyLog
private final boolean traceToStdOut
private boolean daemonDisabled
private final ContextManager ctxMgr
public final boolean skipDisposableStats
Made public to allow access for CreateIndexConstantAction and FromBaseTable, but this is no longer necessary when the debug property to keep disposable statistics is removed.
private LanguageConnectionContext daemonLCC
private final Database db
private final java.lang.String dbOwner
private final java.lang.String databaseName
private final java.util.ArrayList<TableDescriptor> queue
private java.lang.Thread runningThread
private int errorsConsecutive
private long errorsUnknown
private long errorsKnown
private long wuProcessed
private long wuScheduled
private long wuRejectedDup
private long wuRejectedFQ
private long wuRejectedOther
private final long timeOfCreation
private long runTime
private final java.lang.StringBuffer tsb
public IndexStatisticsDaemonImpl(HeaderPrintWriter log, boolean doLog, java.lang.String traceLevel, Database db, java.lang.String userName, java.lang.String databaseName)
log
- the log to write todoLog
- whether to log activity informationtraceLevel
- whether, and to where, trace information should be
written ("off|log|stdout|both")db
- the database ("off|log|stdout|both")userName
- the name of the database ownerdatabaseName
- the name of the database (not stored in the db obj)private boolean dbAtLeast10_9(Database db)
public void schedule(TableDescriptor td)
Assume the descriptor will be valid until we get around to generate the statistics. If it turns out to be invalid, it will be discarded.
schedule
in interface IndexStatisticsDaemon
td
- base table descriptor to update index statistics forprivate boolean acceptWork(TableDescriptor td)
td
- the table descriptor to checktrue
if work can be accepted, false
if not.private void generateStatistics(LanguageConnectionContext lcc, TableDescriptor td) throws StandardException
This method is run as a background task.
lcc
- connection context to use to perform the worktd
- target base table descriptorStandardException
- if accessing the conglomerates failprivate boolean isShuttingDown()
private void updateIndexStatsMinion(LanguageConnectionContext lcc, TableDescriptor td, ConglomerateDescriptor[] cds, boolean asBackgroundTask) throws StandardException
API note: Using null
to update the statistics
for all conglomerates is preferred over explicitly passing an array with
all the conglomerates for the table. Doing so allows for some
optimizations, and will cause a disposable statistics check to be
performed.
lcc
- language connection context used to perform the worktd
- the table to update index stats forcds
- the conglomerates to update statistics for (non-index
conglomerates will be ignored), null
means all indexesasBackgroundTask
- whether the updates are done automatically as
part of a background task or if explicitly invoked by the userStandardException
- if something goes wrongprivate void writeUpdatedStats(LanguageConnectionContext lcc, TableDescriptor td, UUID index, long numRows, long[] cardinality, boolean asBackgroundTask) throws StandardException
lcc
- connection context to use to perform the worktd
- the base tableindex
- the index of the base tablenumRows
- number of rows in the base tablecardinality
- the number of unique values in the index (per number
of leading columns)asBackgroundTask
- whether the update is done automatically as
part of a background task or if explicitly invoked by the userStandardException
- if updating the data dictionary failsprivate void invalidateStatements(LanguageConnectionContext lcc, TableDescriptor td, boolean asBackgroundTask) throws StandardException
lcc
- connection context to use to perform the worktd
- the table to invalidate forasBackgroundTask
- whether the update is done automatically as
part of a background task or if explicitly invoked by the userStandardException
- if the invalidation request failsprivate void setHeapRowEstimate(TransactionController tc, long tableId, long rowEstimate) throws StandardException
tc
- transaction to usetableId
- the heap tablerowEstimate
- estimate of number of rows in the tableStandardException
- if accessing the table failspublic void run()
This method will be run in a separate thread, and it will keep working as long as there is work to do. When the queue is exhausted, the method will exit (the thread dies).
run
in interface java.lang.Runnable
private void processingLoop()
public void runExplicitly(LanguageConnectionContext lcc, TableDescriptor td, ConglomerateDescriptor[] cds, java.lang.String runContext) throws StandardException
runExplicitly
in interface IndexStatisticsDaemon
lcc
- connection context to use to perform the worktd
- the base tablecds
- the indexes to update (non-index conglomerates are ignored)runContext
- the context in which the operation is run (i.e.
'ALTER TABLE', may be null
)StandardException
- if updating the index statistics failspublic void stop()
Will also clear the queue and print runtime statistics to the log the first time the method is invoked.
stop
in interface IndexStatisticsDaemon
private boolean handleFatalErrors(ContextManager cm, StandardException se)
cm
- context managerse
- the exception to handletrue
if the error was handled, false
otherwiseprivate boolean handleExpectedErrors(TableDescriptor td, StandardException se)
The logging of expected errors is for observability purposes only. The daemon is capable of dealing with these errors, and no interaction from the user is expected.
se
- the exception to handletrue
if the error was handled, false
otherwiseprivate boolean handleUnexpectedErrors(TableDescriptor td, StandardException se)
Unexpected errors are error conditions the daemon isn't set up to handle specifically. For this reason the stack trace will be logged to allow for later investigation.
In general it is expected that the daemon will be able to recover by dropping the current unit of work and move on to the next one (if any).
se
- the exception to handletrue
if the error was handled, false
otherwiseprivate static void sleep(long ms)
ms
milliseconds.
No guarantee is provided for the minimum amount of time slept. If interrupted, the interrupt flag will be set again.
ms
- target sleep timeprivate static java.lang.String fmtScanTimes(long[][] timings)
private void log(boolean asBackgroundTask, TableDescriptor td, java.lang.String msg)
private void log(boolean asBackgroundTask, TableDescriptor td, java.lang.Throwable t, java.lang.String msg)
Note that if asBackgroundTask
is false, nothing will be logged
currently.
asBackgroundTask
- true
if logging for the background
daemon automatically updating stats, false
if nottd
- current table descriptor being worked on, may be null
t
- raised error, may be null
msg
- the message to logprivate void logAlways(TableDescriptor td, java.lang.Throwable t, java.lang.String msg)
td
- current table descriptor being worked on, may be null
t
- raised error, may be null
msg
- the message to logprivate void trace(int indentLevel, java.lang.String msg)
private void appendRunStats(java.lang.StringBuffer sb)
sb
- the string buffer to append toprivate static java.lang.String cardToStr(long[] cardinality)
cardinality
- index cardinalityprivate static java.lang.String extractIstatInfo(java.lang.Throwable t)
private static ContextService getContextService()
Apache Derby V10.14 Internals - Copyright © 2004,2018 The Apache Software Foundation. All Rights Reserved.