Run it. Inspect it. Query it.

Measures and formulas

Read the formula, work through an example and find the published fields for Theil, process times, work in progress, survival estimates and other measures.

The measure reference covers the development snapshot Engine 19.0.0rc597 / Reader 9.0.0rc656. The command and setup reference below remains on Engine 18.6.0rc590 / Reader 8.11.0rc649.

Browse the measures →

01 / Commands

Commands

Run the Engine with observatory and the Reader with observatory-reader. Reader options are listed on the installation page.

observatory run [CONFIG]

Reads every source, writes the result file and prints its path. --config FILE names the configuration, --out FILE the result file and --tasks ID … limits the analyses that run.

observatory validate --config FILE

Checks every source against its contract and prints one JSON line per source: its status, the rows read, accepted and rejected, and a note.

observatory inspect FILE --json

Prints row counts for non-sensitive tables. Add --include-sensitive to include every published table.

observatory gui

The desktop launcher. It needs the gui extra. Its options are --config FILE, --db FILE and --check.

Exit codes

Exit code 0 means success. Exit code 2 means the command refused its input and wrote error: with the reason to standard error. A run also exits with 2 when its status begins with BLOCKED_. Validation exits with 2 when a source is neither ACCEPTED nor SKIPPED, unless it failed with required: false and another analytical source supplied admitted rows. A non-zero row_residual also causes exit 2: the physical row count does not equal accepted plus rejected plus skipped rows. Rejected rows alone are not a validation failure. Configure your scheduler to stop on any non-zero exit code.

Environment

OBSERVATORY_PSEUDONYMIZATION_KEY

A 32-byte key, written as 64 hexadecimal characters and required when ANONYMIZE: true. Without ANONYMIZE: true the key is ignored. OBSERVATORY_PSEUDONYMIZATION_KEY_FILE reads it from a file.

OBSERVATORY_MAX_NATIVE_THREADS

A positive integer that caps Polars and BLAS threads.

OBSERVATORY_READER_HOST, OBSERVATORY_READER_PORT

The current Reader does not read these environment variables. Use --host and --port instead.

OBSERVATORY_READER_*

Reader authentication and HTTPS settings: BASIC_AUTH (user:password), BEARER_TOKEN, PROXY_AUTH_HEADER with TRUSTED_PROXIES, REQUIRE_AUTH=1, and ALLOW_HEALTHZ=0 to put /healthz behind authentication.

02 / Configuration

Configuration

One YAML file describes a run. Its top-level keys are version, output, analysis, sources, systems, settings, identity_resolution, calendar_definitions and calendar_policies. An unknown key is refused. The sample’s own file, with all of its sources, is demo/observatory.yaml.

# observatory.yaml
version: 1
output: {directory: out}        # out/observatory.sqlite, beside this file
analysis:
  windows:                      # a unique name; start and end optional
  - {name: march, start: '2026-03-02', end: '2026-04-01'}
  tasks:                        # optional; an id or {id, params}
  - id: process.edges
    params:
      collapsed_path_activities: [Register Claim, Close Claim]
sources:
- id: cases
  path: claims_ops.csv          # relative to this file
  source_system: generic        # a contract id in the catalogue
  role: event_log
  domain: process_flow
  grain: process_event
  authority: primary
  required: true
  system: claims               # namespace for ids; list it under systems
  object_type: claim
  params: {trace_completeness: complete_history}
  field_map:                    # canonical field: source column
    event_id: event_ref
    trace_id: case_id
    activity: activity
    start_ts: started
    end_ts: finished
    employee_id: handled_by
  sequence:                     # the order within a case
    fields: [{column: step_no, type: integer, direction: ascending}]
    basis: case_step_number
settings:
  TIMEZONE: Europe/Stockholm
  TERMINAL_ACTIVITY_TOKENS: [Close Claim]
  SYSTEM_ACTOR_TOKENS: [System]
# …further sources, identity_resolution and calendar_definitions: see demo/observatory.yaml

Source keys

id, path, source_system

The source’s name, its file, and the ID of a contract in the adapter catalogue. The IDs are the names under contracts: in catalog.yaml, and obs.available_yaml_adapters() lists all 114.

role, domain, grain, authority

What the rows are. role, grain and authority must be declared. The sample uses event_log with process_flow, work_interval with workforce_time, object_master with context and desktop_activity with desktop_activity. authority is primary, secondary or context.

required, include

A missing required source blocks the run. include: false skips a source.

system, object_type

The namespace for work-object and account IDs, and the object type of each row.

timezone, encoding, delimiter, locale

Parsing for one source. Without them, the settings apply.

field_map

Each canonical field and the one physical column that holds it.

sequence

The declared order columns and their basis.

params

Contract options such as trace_completeness, duration_evidence_policy and sheet, which selects an Excel worksheet.

object_link_expectation, object_relation_type

object_relation plus a relation type makes the source a set of links between objects.

property_columns

Columns published as typed object attributes: {column: {attribute, type, unit_column, unit}}.

source_context, json_envelope

A declared parent object_id for saved pages, and the exact key paths (rows, errors, has_more) into a JSON document.

Settings

The sample uses these settings: TIMEZONE, SYSTEM_ACTOR_TOKENS, TERMINAL_ACTIVITY_TOKENS, CANCEL_ACTIVITY_TOKENS, QUALITY_PASS_TOKENS and QUALITY_FAIL_TOKENS. WAIT_TARGET_SECONDS and STANDARD_WORK_SECONDS each map an activity to seconds. SOD_RULES lists pairs of activities. REFERENCE_PROCESS is an ordered list of activities. The Engine also accepts REOPEN_ACTIVITY_TOKENS, DATE_FORMAT, INPUT_LOCALE, CSV_EXTRA_FIELD_POLICY, ANONYMIZE and ADAPTERS_DIR. ANONYMIZE is off unless you set it to true; with a key set, each name then becomes a pseudonym.

03 / Before you write a configuration

Before you write a configuration

Lifecycle tokens

TERMINAL_ACTIVITY_TOKENS and its relatives match the activity, not raw status values.

Work-time evidence

A source’s rows count as self-reported work unless they state source_metric_basis: source_event_stream or come from a desktop export, which is system-observed. Processing time is never a person’s work time.

Order

Order is declared with sequence. Timestamps are never promoted to process order, so without a sequence the Engine keeps the events and withholds paths and waits.

Calendars

Without a declared calendar or a workforce_schedule source, schedule columns are empty with a warning, never zero. The most specific assignment wins: employee, then team, then organisation.

The first and last day

A schedule counts only inside the run’s analysis window. With no analysis.windows the window is the observed period, so the first and last day count only part of the day. For whole days, set a window that covers them.

People

An account that is not listed under identity_resolution stays scoped to its own system. List a person’s accounts to join them across systems.

Time zones

Zones are IANA names such as Europe/Stockholm. A clock value with no zone is read in the run’s TIMEZONE, and the run says so.

04 / Python API

Python API

The command line is a thin shell over this API, so a script can do everything the commands do. Programs and language models read the result through it, or open the file with SQL.

import observatory as obs

# run, validate and inspect
path = obs.run_pipeline("observatory.yaml")
obs.validate_sources("observatory.yaml")
obs.inspect_database(path, include_sensitive=True)
obs.available_yaml_adapters()

# read a result file
results = obs.open_results(path, include_sensitive=True)
results.select("duration_stats", where={"subject_kind": "activity"})
results.query_readonly("SELECT ...", allowed_tables=["objects"])

# run in memory
ctx = obs.ProcessContext.from_config("observatory.yaml")
plan = ctx.plan(outputs=["objects"])
result = ctx.run(
    outputs=["objects"], windows=["all"], include_sensitive=True
)
result.frame("objects")

select returns a list of rows. query_readonly and frame return Polars frames. Sensitivity is per table, and most tables are sensitive. Without include_sensitive=True the API lists only analysis_windows, flow_days, rules and run. A call for any other table raises SensitiveOutputError, even when you select no person. The flag guards the API, not the file: a SQLite client reads every table.

query_readonly runs one read-only statement. It refuses a table you did not list, a statement that writes, more than one statement, NATURAL JOIN and JOIN … USING. Write every join with an explicit ON, because a join with none runs as a cross join. Its errors derive from ObservatoryApiError: ApiInputError, SensitiveOutputError, StoredResultsError and ReadOnlyQueryError.

A call to select or query_readonly returns at most 1,000 rows unless it sets limit=, which accepts up to 100,000. At the limit, select stops without a message, and query_readonly raises ReadOnlyQueryError when the result is larger. It also stops a result above 64 MiB and a statement that runs longer than 30 seconds.

05 / Warnings

Warnings

A completed run writes the warnings and run tables. Its run_status is OK or OK_WITH_WARNINGS; informational records alone do not make the status a warning. A blocked run writes no result file.

Severity high means rows or authority were refused or withheld. warning means something was dropped, assumed or left unknown. info is a scope note. These are the warnings you meet first. The package lists more, and a release check fails when a fixture or demo run raises a warning that is not listed.

input_columns_unmappedwarning

The file has columns the contract does not read. The message lists them.

What to do: If one holds a field you need, add its name to that field’s column_map list in your adapters folder. Otherwise ignore it.

invalid_timestamphigh

Rows with timestamps that do not match the declared formats were rejected.

What to do: Add the file’s format to timestamp_formats, or fix the export format.

end_before_starthigh

Rows with an end time earlier than their start time were rejected.

What to do: Check the source’s clocks.

timestamp_zone_from_run_settingwarning

Clock values had no zone, so they were read in the run’s TIMEZONE.

What to do: Set TIMEZONE, or export with a zone.

duplicate_event_grain_consolidatedwarning

The same event was exported more than once. Extra copies are listed as exact_duplicate_event in row_decisions and the event is counted once.

What to do: Nothing, unless the copies are distinct events. Then export an event ID.

sequence_withheldinfo

Events were kept but have no declared order. Timestamps are never promoted to process order.

What to do: Declare sequence.fields and sequence.basis.

person_identity_source_scopedwarning

People are known only inside their own system and are not joined across systems.

What to do: List them in identity_resolution.canonical_people to join them.

work_record_without_durationwarning

Work records state no work time, so every total that includes them is unknown, not smaller.

What to do: Export the duration.

schedule_clipped_to_observed_periodinfo

A window with no explicit start or end clips each schedule to the observed period. On the first and last day, scheduled_sec covers only part of the day.

What to do: Set analysis.windows start and end to whole days.

terminal_vocabulary_unmatchedwarning

No configured completion or cancellation activity matched any work object. All objects remain open, so no lead time is measured.

What to do: Name the completion activity in TERMINAL_ACTIVITY_TOKENS and the cancellation activity in CANCEL_ACTIVITY_TOKENS. No change is needed for work objects that are genuinely still open.

time_to_close_insufficient_objectswarning

Too few closed objects are available to interpret the time-to-close curve as a population estimate. The message gives the count.

What to do: Treat the curve as descriptive, or include a wider time window or more sources.

06 / Adapter contracts

Adapter contracts

A contract maps source headers to canonical fields. To support a different header, edit a copy of the contract in your own folder. Leave the packaged catalogue unchanged.

1. Copy engine/observatory/catalog.yaml from the package into a folder inside the folder that holds your configuration.
2. In the contract, add the new header to the field’s column_map list. If the field is required, name the new header in required_fields in place of the old one.
3. Set ADAPTERS_DIR under settings to that folder. A path outside the configuration’s folder is refused.

The file must keep catalog_version and engine unchanged and stay under 1,048,576 bytes. This is the whole change for a Jira worklog export whose person column is called authorId:

jira:
  column_map:
-   employee_id: [author.accountId]
+   employee_id: [author.accountId, authorId]
  …
-   required_fields: [issueId, …, author.accountId]
+   required_fields: [issueId, …, authorId]

$ observatory validate --config observatory.yaml
{"adapter": "jira", …, "status": "FAILED",
 "note": "… required field(s) unavailable: ['author.accountId']"}
# after the change, with settings: ADAPTERS_DIR: adapters
$ observatory validate --config observatory.yaml
{"accepted": 3, "adapter": "jira", …, "rejected": 0, … "status": "ACCEPTED"}

YAML configures existing operations. New parsing behaviour requires Engine development, available through consulting.

07 / The result file

The result file

The schema document lists every table and column with its kind, how it aggregates and what it means. It is generated from the declaration that the Engine publishes against, and a release check fails when the two differ. The current result contract contains tables only. The published sample from Engine 18.6.0rc450 also includes the three older views bi_objects, bi_transitions and bi_slas.

Three things to know before you query

Select one scope

A scope_id is one analysis window and one evidence stream. The same object or person appears in several scopes, so select one scope before you count or sum.

Keep evidence classes separate

self_reported is reported_work, system_observed is observed_work and unclassified is unclassified_work. Classes overlap in time and are never summed. processing_time is the span of a process step and has no evidence class.

Completed is not approved

An object whose outcome is completed reached the end of its history, whatever the business result. An approved claim and a rejected claim both end completed.