Diagnostics (distributed)¶
The Dask distributed scheduler provides feedback in two forms:
- A progress bar suitable for interactive use in consoles or notebooks
- An interactive dashboard, containing several plots and tables with live information
Progress bar¶
progress (*futures, **kwargs) |
Track progress of futures |
The dask.distributed progress bar differs from the ProgressBar
used for
local diagnostics.
The progress
function takes a Dask object that is executing in the background.
# Single machine progress bar
from dask.diagnostics import ProgressBar
with ProgressBar():
x.compute()
# Distributed scheduler ProgressBar
from dask.distributed import Client, progress
client = Client() # use dask.distributed by default
x = x.persist() # start computation in the background
progress(x) # watch progress
x.compute() # convert to final result when done if desired
Dashboard¶
Client ([address, loop, timeout, …]) |
Connect to and drive computation on a distributed Dask cluster |
If Bokeh is installed then the dashboard will start up automatically whenever the scheduler is created. For local use this happens automatically when you create a client with no arguments
from dask.distributed import Client
client = Client() # start distributed scheduler locally. Launch dashboard
It is typically served at http://localhost:8787/status , but may be served elsewhere if this port is taken. The address of the dashboard will be displayed if you are in a Jupyter Notebook.
There are numerous pages with information about task runtimes, communication, statistical profiling, load balancing, memory use, and much more. For more information we recommend the following video guide:
External Documentation¶
More in-depth technical documentation about Dask’s distributed scheduler is available at https://distributed.dask.org/en/latest
API¶
-
dask.distributed.
progress
(*futures, **kwargs)¶ Track progress of futures
This operates differently in the notebook and the console
- Notebook: This returns immediately, leaving an IPython widget on screen
- Console: This blocks until the computation completes
Parameters: futures: Futures
A list of futures or keys to track
notebook: bool (optional)
Running in the notebook or not (defaults to guess)
multi: bool (optional)
Track different functions independently (defaults to True)
complete: bool (optional)
Track all keys (True) or only keys that have not yet run (False) (defaults to True)
Notes
In the notebook, the output of progress must be the last statement in the cell. Typically, this means calling progress at the end of a cell.
Examples
>>> progress(futures) [########################################] | 100% Completed | 1.7s