Entities

HWM

Description

High Water Mark (or HWM for short) is a record which allows tracking state of operations (usually read). For example:

• Save max value of a column read from a table (e.g. updated_at), and then use it exclude already read rows on the next read.
• Save list of files handled by a process, and then use it to exclude these files on the next read.
• Same max modification time of files handled by a process, and then use it to exclude these files on the next read.

Each HWM record is bound to a specific Namespace. HWM name is unique within this Namespace.

Users may create unlimited number of HWMs within a namespace.

Fields

HWM record has following fields:

• id: integer - internal HWM identifier, generated using sequence, read only.
• namespace_id: integer - bound to a Namespace, mandatory.
• name: string - unique name within a Namespace, mandatory.
• value: json - value associated with HWM, mandatory.
• type: string - any string describing type of value field content, mandatory.
• description: string - human readable description.
• entity: string | null - name of entity (e.g. table name, folder) which value is bound to.
• expression: string | null - expression (e.g. column name) used to get value from entity.
• changed_at: datetime - filled up automatically on each item change, read only.
• changed_by: User | null - filled up automatically on each item change, read only.

type and value can contain any value, and are not parsed or checked by backend. This allows users to create HWMs of any type in any time, without patching backend. But it is up to user to keep consistency of these fields.

Limitations

HWM cannot be moved between namespaces. Users have to explicitly create a copy of existing HWM in new namespace, and delete old HWM.

Namespace

Description namespace

Namespace is a container for HWM records.

Each namespace has an owner which can alter namespace properties, see Role Permissions.

Users may create unlimited number of namespaces. If user created a namespace, it is automatically set as the namespace owner.

Fields namespace

Namespace record has following fields:

• id: integer - internal namespace identifier, generated using sequence, read only.
• name: string - unique per Horizon instance, mandatory
• description: string - human readable description.
• owned_by: User - is set automatically while namespace is created, mandatory.
• changed_at: datetime - filled up automatically on each item change, read only.
• changed_by: User | null - filled up automatically on each item change, read only.

User

Description user

Users are used for:

• Authentication
• RBAC permissions model
• Keeping track of changes made on Namespace or HWM.

User records are created automatically after successful authentication.

Fields user

User record has following fields:

• id: integer - internal user identifier, generated using sequence, read only
• username: string - unique per Horizon instance
• is_admin: boolean - flag for SUPERUSER role.

Limitations user

For now it is not possible to remove user after creation.

Permission

Description permission

User can be assigned a Role within a Namespace, which can allow or disallow performing specific operation within this Namespace. User can have different roles in different namespaces. See Role Permissions.

Fields permission

Permission record has following fields:

• user: User
• namespace: Namespace
• role: enum.

Limitations permission

User can be assigned only one Role within a Namespace.

NamespaceHistory

Description namespacehistory

Change of each Namespace value produces a HWMHistory item, which can be used for audit purpose. History is append-only, items cannot be changed or deleted using API.

Fields namespacehistory

NamespaceHistory record has following fields (all read-only):

• id: integer - internal history item identifier, generated using sequence.
• namespace_id: integer - bound to Namespace item.
• name: string.
• description: string.
• owned_by: User.
• changed_at: datetime - filled up automatically on each item change.
• changed_by: User | null - filled up automatically on each item change.
• action: string - change description, e.g. Created, Updated, Deleted.

HWMHistory

Description HWMhistory

Change of each HWM value produces a HWMHistory item, which can be used for audit purpose. History is append-only, items cannot be changed or deleted using API.

Fields HWMhistory

HWMHistory record has following fields (all read-only):

• id: integer - internal history item identifier, generated using sequence.
• hwm_id: integer - bound to HWM item.
• name: string.
• value: any | null.
• type: string.
• description: string.
• entity: string | null.
• expression: string | null.
• changed_at: datetime - filled up automatically on each item change.
• changed_by: User | null - filled up automatically on each item change.
• action: string - change description, e.g. Created, Updated, Deleted.

Entity Diagram

---
title: Entity Diagram
---
erDiagram
    direction LR
    User {
        number id
        string username
        bool is_admin
    }

    Namespace {
        number id
        number namespace_id
        string name
        string owned_by
        string description
        string changed_at
        string changed_by
    }

    HWM {
        number id
        string name
        string type
        string value
        string description
        string entity
        string expression
        string changed_at
        string changed_by
    }

    NamespaceHistory {
        number id
        number namespace_id
        string name
        string owned_by
        string description
        string changed_at
        string changed_by
        string action
    }


    HWMHistory {
        number id
        number hwm_id
        number namespace_id
        string name
        string type
        string value
        string description
        string entity
        string expression
        string changed_at
        string changed_by
        string action
    }

    Permission {
        number user_id
        number namespace_id
        string role
    }

    HWM ||--o{ Namespace: contains
    Namespace }o--o| NamespaceHistory: contains
    HWM }o--o| HWMHistory: contains
    Namespace  ||--o{ User: contains
    Namespace }o--|| Permission: contains
    Permission ||--o{ User: contains