Engine 19.0.0rc597 · Reader 9.0.0rc656 · Development snapshot
Kernel density estimation, or KDE, turns recorded values into a smooth estimate of their distribution. Ghost Observatory fits Gaussian kernels to the natural logarithms of positive durations. The resulting curve shows where durations cluster on a logarithmic time axis.
Use it when a median leaves an unanswered question: do most observations form one group, or does the distribution have a long tail or several concentrations? Read the raw observations and sample count alongside the curve. The bandwidth can smooth away features, especially in small populations.
The Gaussian KDE formula
For m positive durations ti, set yi = ln(ti). Each observation contributes an equally weighted Gaussian kernel with width h.
f̂Y(y) = [1 / (m h √(2π))] ∑i=1m exp[−½ ((y − yi) / h)2]
The vertical unit is density per unit of ln(seconds). A height of 0.5 does not mean that half the observations have that duration. Probability belongs to an interval under the density. A density per second would require dividing the log-density value by t; the published arrays retain the log-density scale.
Scott and Silverman bandwidths
The bandwidth controls the amount of smoothing. The Engine first calculates the sample standard deviation s of the log durations, using denominator m − 1. It then applies the configured factor.
| Method | Kernel width |
|---|---|
| Scott, the default | h = s m−1/5 |
| Silverman | h = s (3m / 4)−1/5 |
This Silverman option is the dimension-one form of the multivariate bandwidth factor. It does not use the interquartile-range adjustment sometimes called Silverman’s univariate rule of thumb.
An activity fits its own bandwidth. For an eligible person or team work-time curve, the Engine uses the larger of the activity’s width and the width fitted to that subject’s observations. The comparison population uses that same width, so the comparison does not introduce a different smoothing setting.
A worked example with 20 durations
Take ti = 10 exp(i / 10) seconds for i from 0 to 19: twenty durations from 10 to about 66.859 seconds. Applying the bandwidth and Gaussian-density formulas gives the values below. The density is evaluated at the geometric mean of the durations; this lies between two coordinates of the published grid.
| Quantity | Result |
|---|---|
| Sample standard deviation of log durations | 0.591608 |
| Scott bandwidth | 0.324959 |
| Silverman bandwidth | 0.344204 |
| Default Scott grid | 48 coordinates, from 3.772 to 177.232 seconds |
| Density at 25.857 seconds | 0.498997 per unit of ln(seconds) |
The grid extends three activity bandwidths beyond each end of the observed log sample. Its wider range reflects the Gaussian kernels’ tails. It does not assert that an observation occurred at either endpoint.
Which observations enter a curve?
Populations stay separate by analysis scope, activity and duration source. Processing times form their own population; self-reported, system-recorded and unclassified work durations each form another. An observation has equal weight within its population. A person with more admitted records contributes more observations.
Processing-time curves are calculated per activity. The Engine also calculates person and team curves for eligible work records. A subject’s comparison contains every other duration in the same activity and source, including records with no attributable person. Both sides need the configured minimum number of positive observations.
By default, the curve task selects the 12 activities with the most admitted durations. Activity and comparison estimates use at most 10,000 evenly spaced sorted observations when their populations are larger. The stored curve count still describes the full positive population. This deterministic selection is separate from the underlying count.
Person and team selection has additional limits: by default, at most 160 curves of each kind, and no subject curves when an activity/source population exceeds 50,000 observations. These settings can change. A dated analysis window excludes a duration that crosses its boundary.
KDE versus a histogram
A histogram groups observations into bins; changing the bin edges can change its appearance. A kernel density estimate places a smooth contribution around each observation and adds those contributions. Its bandwidth plays the main smoothing role: a small bandwidth exposes local detail, while a large one can merge distinct concentrations.
Neither view makes the sample representative. For a duration comparison, keep the population, time unit and transformation explicit. The log-duration KDE described here is not the same curve as fitting a KDE directly to seconds. The SciPy method reference explains why bandwidth selection matters.
Stored results and the current Reader view
The Engine publishes coordinates in duration_stats.duration_sec_json, their matching densities in kde_density_json, and the subject’s comparison in comparison_kde_density_json. It also stores kde_bandwidth, curve_sample_n, comparison_sample_n and curve_status.
Timing → Processing time curves reads those arrays and draws them on a log time axis. The Reader uses one density-height scale and adds the stored median and P90. It does not fit a replacement curve in the browser.
That dedicated view currently requests processing time. Person/team work curves are Engine outputs, rather than a separate work-curve view in this Reader. A person’s processing-time row carries activity_only: select the activity to inspect its curve.
When a curve is withheld
The default minimum is 20 positive observations and can be raised. Zero durations remain eligible for ordinary duration summaries but cannot enter a logarithm. If the positive log durations are equal within 10−12, the status is all_equal and no automatically fitted curve is published.
Other statuses distinguish too few observations, too few comparison observations, selection limits, missing person authority and numerical failures. Read curve_status before interpreting a blank curve. Ordinary sample_n can include zero durations, whereas curve_sample_n counts the positive population.
Related measures and sources
Use median and percentiles for specific positions in the observed distribution, and control charts to retain chronological order. See processing time for the duration itself.
Method reference: the official SciPy Gaussian KDE documentation explains the Scott and Silverman factors and bandwidth conventions. Ghost Observatory implements its calculations in its own code.