Entities

HWM

Description

High Water Mark (or HWM for short) is a record which allows tracking state of operations (usually read). For example:

• Save max value of a column read from a table (e.g. updated_at), and then use it exclude already read rows on the next read.

• Save list of files handled by a process, and then use it to exclude these files on the next read.

• Same max modification time of files handled by a process, and then use it to exclude these files on the next read.

Each HWM record is bound to a specific Namespace. HWM name is unique within this Namespace.

Users may create unlimited number of HWMs within a namespace.

Fields

HWM record has following fields:

• id: integer - internal HWM identifier, generated using sequence, read only.

• namespace_id: integer - bound to a Namespace, mandatory.

• name: string - unique name within a Namespace, mandatory.

• value: json - value associated with HWM, mandatory.

• type: string - any string describing type of value field content, mandatory.

• description: string - human readable description.

• entity: string | null - name of entity (e.g. table name, folder) which value is bound to.

• expression: string | null - expression (e.g. column name) used to get value from entity.

• changed_at: datetime - filled up automatically on each item change, read only.

• changed_by: User | null - filled up automatically on each item change, read only.

type and value can contain any value, and are not parsed or checked by backend. This allows users to create HWMs of any type in any time, without patching backend. But it is up to user to keep consistency of these fields.

Limitations

HWM cannot be moved between namespaces. Users have to explicitly create a copy of existing HWM in new namespace, and delete old HWM.

Namespace

Description

Namespace is a container for HWM records.

Each namespace has an owner which can alter namespace properties, see Role Permissions.

Users may create unlimited number of namespaces. If user created a namespace, it is automatically set as the namespace owner.

Fields

Namespace record has following fields:

• id: integer - internal namespace identifier, generated using sequence, read only.

• name: string - unique per Horizon instance, mandatory

• description: string - human readable description.

• owned_by: User - is set automatically while namespace is created, mandatory.

• changed_at: datetime - filled up automatically on each item change, read only.

• changed_by: User | null - filled up automatically on each item change, read only.

User

Description

Users are used for:

• Authentication

• RBAC permissions model

• Keeping track of changes made on Namespace or HWM.

User records are created automatically after successful authentication.

Fields

User record has following fields:

• id: integer - internal user identifier, generated using sequence, read only

• username: string - unique per Horizon instance

• is_active: boolean - flag if user is allowed to log in.

• is_admin: boolean - flag for SUPERUSER role.

Limitations

For now it is not possible to remove user after creation.

Permission

Description

User can be assigned a Role within a Namespace, which can allow or disallow performing specific operation within this Namespace. User can have different roles in different namespaces. See Role Permissions.

Fields

Permission record has following fields:

• user: User

• namespace: Namespace

• role: enum.

Limitations

User can be assigned only one Role within a Namespace.

NamespaceHistory

Description

Change of each Namespace value produces a HWMHistory item, which can be used for audit purpose. History is append-only, items cannot be changed or deleted using API.

Fields

NamespaceHistory record has following fields (all read-only):

• id: integer - internal history item identifier, generated using sequence.

• namespace_id: integer - bound to Namespace item.

• name: string.

• description: string.

• owned_by: User.

• changed_at: datetime - filled up automatically on each item change.

• changed_by: User | null - filled up automatically on each item change.

• action: string - change description, e.g. Created, Updated, Deleted.

HWMHistory

Description

Change of each HWM value produces a HWMHistory item, which can be used for audit purpose. History is append-only, items cannot be changed or deleted using API.

Fields

HWMHistory record has following fields (all read-only):

• id: integer - internal history item identifier, generated using sequence.

• hwm_id: integer - bound to HWM item.

• name: string.

• value: any | null.

• type: string.

• description: string.

• entity: string | null.

• expression: string | null.

• changed_at: datetime - filled up automatically on each item change.

• changed_by: User | null - filled up automatically on each item change.

• action: string - change description, e.g. Created, Updated, Deleted.

Entity Diagram