1 Commits

9 changed files with 2751 additions and 1504 deletions
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+738 -216
View File
File diff suppressed because it is too large Load Diff
+279 -112
View File
@@ -7,11 +7,79 @@
@prefix pr: <https://ontology.pernod-ricard.com/metamodel/> . @prefix pr: <https://ontology.pernod-ricard.com/metamodel/> .
################################################################# #################################################################
# PERNOD RICARD DATA META MODEL - T-Box v1.2 # PERNOD RICARD DATA META MODEL - T-Box v1.5
# Owner: Bastien Gourdon - Data Governance Office # Owner: Bastien Gourdon - Data Governance Office
# #
# WHAT CHANGED vs v1.1 # WHAT CHANGED vs v1.4
# -------------------- # --------------------
# CuratedObject -> DefinedObject, ObservedObject -> CapturedObject, in the
# IRIs and not only in the labels. v1.4 renamed what screens showed and left
# the terms themselves untouched, so the model said one thing and the
# interface another -- exactly the split the short-label work was meant to
# end. Every reference follows: subclass declarations, the disjointness
# axiom, the shapes and their messages, and the prose in the comments.
#
# No instance changes. Nothing in the A-Box ever named these two classes
# directly: an object reaches them through its layer, by entailment. The old
# IRIs are kept deprecated per LC-002 so a domain file written against them
# still classifies.
#
# WHAT CHANGED vs v1.3 (kept for reference)
# -----------------------------------------
# A. PhysicalRelation -> DataStructure. "Relation" meant a table in Codd's
# sense, which reads as a property in an ontology and locks the class into
# the tabular world. A Neo4j node label, a Mongo collection and a Kafka
# topic are data structures; none of them is a relation. The class is also
# the declared EXTENSION POINT for non-tabular storage -- see its comment
# for the four lines that would insert Dataset and Graph beneath it.
#
# B. hasSource -> sourcedFrom, pointing at a System instead of holding a
# string. "Which objects depend on MDM Product?" was unanswerable because
# MDM Product was text. It also settles the naming collision the string
# created with BIDataSource: one names a system, the other a BI surface,
# and now they are different kinds of thing rather than two readings of
# one property.
#
# C. storedIn serves two granularities, like represents before it:
# DataObject storedIn DataStructure
# DataElement storedIn Field
# materializedAs is therefore redundant and deprecated. The seam between
# what governance defines and what a harvester captures now counts ONE
# relation instead of two.
#
# D. inSchema moves from DataStructure to nothing declared, and must stay
# OPTIONAL in the shapes. Neo4j has no schema and BigQuery calls the level
# a dataset; a minCount 1 would break on the first non-Snowflake harvest.
#
# WHAT CHANGED vs v1.2 (kept for reference)
# -----------------------------------------
# A. references -> qualifiedBy. "References" named the mechanism, a foreign
# key; qualifiedBy names the reason: the Sell Out dataset is qualified by
# Product, Customer, Currency and Date. It reads without explanation, and
# it is what gives a fact its grain and its analysis context.
#
# B. primaryLocation, a SUB-PROPERTY of storedIn. One Data Element can sit in
# several Fields, one authoritative and the others copies. Two parallel
# properties would be free to disagree -- a field marked primary in one and
# missing from the other. As a sub-property, entailment guarantees the
# primary location is always among the locations. Primary/replica rather
# than master/slave, which the industry has moved away from and which
# reads badly to a business audience.
#
# C. hasShortLabel. rdfs:label carries the exhaustive name (Data Domain, Sell
# Out Data Hub - Gold), hasShortLabel the one screens use (Domain, SODH
# Gold). Same object was being called two things depending on where you
# looked; now both names are declared and the viewers stop hard-coding a
# display dictionary of their own.
#
# D. hasSource is now forbidden on a Business Object as well as on a Business
# Concept. Same argument: a Business Object is a governance grouping, not
# data. Its data lives in the Data Objects that represent it, and those
# carry the system. In v0.6 the 15 objects carried a grab-bag -- physical
# locations, migration notes, arbitration remarks -- under one property.
#
# WHAT CHANGED vs v1.1 (kept for reference)
# -----------------------------------------
# A. physicalizedIn -> storedIn. "Physicalized in" is jargon; the relation # A. physicalizedIn -> storedIn. "Physicalized in" is jargon; the relation
# says where the data sits, and "stored in" says that to anyone. # says where the data sits, and "stored in" says that to anyone.
# #
@@ -103,7 +171,7 @@
# one Product Owner: it is a consumable, and there must be no ambiguity # one Product Owner: it is a consumable, and there must be no ambiguity
# about who runs it. # about who runs it.
# #
# G. realizesMetric KEPT AS A TRANSITIONAL CONTROL # G. exposesMetric KEPT AS A TRANSITIONAL CONTROL
# Declared, required on certified BI fields, and that is all. No rule is # Declared, required on certified BI fields, and that is all. No rule is
# written against the lineage chain: making the assertion conditional on # written against the lineage chain: making the assertion conditional on
# intermediate mappings existing would create confusion and discourage # intermediate mappings existing would create confusion and discourage
@@ -122,11 +190,11 @@
# freeze the term namespace and version the ontology document. # freeze the term namespace and version the ontology document.
# MIGRATION: sed -i 's|/metamodel/0.9/|/metamodel/|g' instances/*.ttl # MIGRATION: sed -i 's|/metamodel/0.9/|/metamodel/|g' instances/*.ttl
# #
# 2. NEW ORTHOGONAL AXIS: CuratedObject vs ObservedObject. # 2. NEW ORTHOGONAL AXIS: DefinedObject vs CapturedObject.
# LAYER says WHAT an object is. PROVENANCE says HOW IT GOT HERE. # LAYER says WHAT an object is. PROVENANCE says HOW IT GOT HERE.
# - CuratedObject : asserted by governance, edited via pull request, # - DefinedObject : written by governance, edited via pull request,
# subject to the full ownership shape. # subject to the full ownership shape.
# - ObservedObject: harvested from a running system, rebuilt every night, # - CapturedObject: read from a running system, rebuilt every night,
# NEVER hand-edited, NOT required to carry owningDomain. # NEVER hand-edited, NOT required to carry owningDomain.
# Without this split, the SHACL ownership shape would demand an owning Data # Without this split, the SHACL ownership shape would demand an owning Data
# Domain and a named owner on every one of the ~50 000 Snowflake columns. # Domain and a named owner on every one of the ~50 000 Snowflake columns.
@@ -136,7 +204,7 @@
# Transformation, plus column-level lineage (derivedFrom). # Transformation, plus column-level lineage (derivedFrom).
# #
# 4. NEW CONSUMPTION LAYER (Tableau + Looker). # 4. NEW CONSUMPTION LAYER (Tableau + Looker).
# BIWorkspace / BIDataSource / BIField / BIReport, plus realizesMetric, # BIWorkspace / BIDataSource / BIField / BIReport, plus exposesMetric,
# which closes the loop between a certified Metric and its actual # which closes the loop between a certified Metric and its actual
# implementations in the BI estate. # implementations in the BI estate.
# #
@@ -151,13 +219,13 @@
<https://ontology.pernod-ricard.com/metamodel/> <https://ontology.pernod-ricard.com/metamodel/>
a owl:Ontology ; a owl:Ontology ;
owl:versionIRI <https://ontology.pernod-ricard.com/metamodel/1.2> ; owl:versionIRI <https://ontology.pernod-ricard.com/metamodel/1.5> ;
owl:priorVersion <https://ontology.pernod-ricard.com/metamodel/1.1> ; owl:priorVersion <https://ontology.pernod-ricard.com/metamodel/1.4> ;
dcterms:title "Pernod Ricard Data Meta Model" ; dcterms:title "Pernod Ricard Data Meta Model" ;
dcterms:creator "Bastien Gourdon" ; dcterms:creator "Bastien Gourdon" ;
dcterms:modified "2026-07-27"^^xsd:date ; dcterms:modified "2026-07-30"^^xsd:date ;
owl:versionInfo "1.2.0-draft-2026-07-27" ; owl:versionInfo "1.5.0-draft-2026-07-30" ;
rdfs:comment """OWL T-Box. Six layers (Ownership, Business, Logical, Delivery, Physical, Consumption) crossed with a provenance axis (Curated vs Observed). Curated objects are asserted by governance and reviewed; Observed objects are harvested from Snowflake, Tableau and Looker and are rebuilt, never edited. Bridge annotation properties are declared for future alignment with domain ontologies.""" . rdfs:comment """OWL T-Box. Six layers (Ownership, Business, Logical, Delivery, Physical, Consumption) crossed with a provenance axis (Defined vs Captured). Defined objects are written by governance and reviewed; Captured objects are read from Snowflake, Tableau and Looker and are rebuilt, never edited. Bridge annotation properties are declared for future alignment with domain ontologies.""" .
################################################################# #################################################################
@@ -187,11 +255,11 @@ pr:denotesActivity a owl:AnnotationProperty ;
# ---- Operational documentation -------------------------------- # ---- Operational documentation --------------------------------
pr:harvestSource a owl:AnnotationProperty ; pr:harvestSource a owl:AnnotationProperty ;
rdfs:label "harvest source" ; rdfs:label "harvest source" ;
rdfs:comment "For Observed terms: the system view or API that populates this term. Documents the contract between the meta model and the harvester." . rdfs:comment "For Captured terms: the system view or API that populates this term. Documents the contract between the meta model and the harvester." .
pr:curationMode a owl:AnnotationProperty ; pr:authoringMode a owl:AnnotationProperty ;
rdfs:label "curation mode" ; rdfs:label "authoring mode" ;
rdfs:comment "ASSERTED (governance writes it), HARVESTED (a job writes it) or PROPOSED_CONFIRMED (a job proposes, a steward confirms)." . rdfs:comment "Who writes this, in practice: ASSERTED (a person does), HARVESTED (a job does) or PROPOSED_CONFIRMED (a job proposes and a person accepts). Renamed from curationMode along with the classes: 'curation' belonged to the same jargon family as 'curated'." .
################################################################# #################################################################
@@ -200,79 +268,106 @@ pr:curationMode a owl:AnnotationProperty ;
pr:MetaModelObject a owl:Class ; pr:MetaModelObject a owl:Class ;
rdfs:label "Meta Model Object" ; rdfs:label "Meta Model Object" ;
rdfs:comment "Anything the meta model describes. Actors are deliberately NOT Meta Model Objects." . rdfs:comment "Anything the meta model describes. Actors are deliberately NOT Meta Model Objects." ;
pr:hasShortLabel "Meta model object" .
pr:CuratedObject a owl:Class ; rdfs:subClassOf pr:MetaModelObject ; pr:DefinedObject a owl:Class ; rdfs:subClassOf pr:MetaModelObject ;
rdfs:label "Curated Object" ; rdfs:label "Defined Object" ;
pr:curationMode "ASSERTED" ; pr:hasShortLabel "Defined" ;
rdfs:comment "Asserted by governance, edited through pull request on the domain TTL file, subject to the full ownership and lifecycle shape (identifier, name, owner, owning Data Domain, status, version)." . pr:authoringMode "ASSERTED" ;
rdfs:comment "Somebody decided it. Written by governance, edited through a pull request on the domain TTL file, and subject to the full identity and lifecycle shape: identifier, name, owner, owning Data Domain, status, version." .
pr:ObservedObject a owl:Class ; rdfs:subClassOf pr:MetaModelObject ; pr:CuratedObject a owl:Class ;
rdfs:label "Observed Object" ; rdfs:label "Curated Object (deprecated)" ;
pr:curationMode "HARVESTED" ; owl:deprecated true ; dcterms:isReplacedBy pr:DefinedObject ;
rdfs:comment "Harvested from a running system and rebuilt on every run. Never hand-edited. Carries sourceIdentifier, fullyQualifiedName and harvestedOn instead of curated ownership: attribution is INFERRED through the Data Element it is mapped to, not asserted on the node." . rdfs:comment "v1.4 term. Renamed because 'curated' is governance jargon; 'defined' says the same thing to a business reader. No instance ever named this class directly." .
pr:OwnershipLayerObject a owl:Class ; rdfs:subClassOf pr:CuratedObject ; pr:CapturedObject a owl:Class ; rdfs:subClassOf pr:MetaModelObject ;
rdfs:label "Captured Object" ;
pr:hasShortLabel "Captured" ;
pr:authoringMode "HARVESTED" ;
rdfs:comment "Read from a running system and rebuilt on every run. Never hand-edited. Carries sourceIdentifier, fullyQualifiedName and harvestedOn instead of declared ownership: attribution is INFERRED through the Data Element it is mapped to, never asserted on the node." .
pr:ObservedObject a owl:Class ;
rdfs:label "Observed Object (deprecated)" ;
owl:deprecated true ; dcterms:isReplacedBy pr:CapturedObject ;
rdfs:comment "v1.4 term. Renamed because 'observed' is passive and technical; 'captured' says where it came from. No instance ever named this class directly." .
pr:OwnershipLayerObject a owl:Class ; rdfs:subClassOf pr:DefinedObject ;
rdfs:label "Ownership & Categorization Layer Object" . rdfs:label "Ownership & Categorization Layer Object" .
pr:BusinessLayerObject a owl:Class ; rdfs:subClassOf pr:CuratedObject ; pr:BusinessLayerObject a owl:Class ; rdfs:subClassOf pr:DefinedObject ;
rdfs:label "Business Layer Object" . rdfs:label "Business Layer Object" .
pr:LogicalLayerObject a owl:Class ; rdfs:subClassOf pr:CuratedObject ; pr:LogicalLayerObject a owl:Class ; rdfs:subClassOf pr:DefinedObject ;
rdfs:label "Logical Layer Object" ; rdfs:label "Logical Layer Object" ;
rdfs:comment "How the data is structured." . rdfs:comment "How the data is structured." .
pr:DeliveryLayerObject a owl:Class ; rdfs:subClassOf pr:CuratedObject ; pr:DeliveryLayerObject a owl:Class ; rdfs:subClassOf pr:DefinedObject ;
rdfs:label "Delivery Layer Object" ; rdfs:label "Delivery Layer Object" ;
rdfs:comment "What we promise, to whom, until when. A different question from how the data is structured, hence a layer of its own: version, owner, changelog, support channel, consumers to warn." . rdfs:comment "What we promise, to whom, until when. A different question from how the data is structured, hence a layer of its own: version, owner, changelog, support channel, consumers to warn." .
pr:PhysicalLayerObject a owl:Class ; rdfs:subClassOf pr:ObservedObject ; pr:PhysicalLayerObject a owl:Class ; rdfs:subClassOf pr:MetaModelObject ;
rdfs:label "Physical Layer Object" . rdfs:label "Physical Layer Object" ;
pr:ConsumptionLayerObject a owl:Class ; rdfs:subClassOf pr:ObservedObject ; rdfs:comment """The one layer whose members do NOT share a provenance, which is why this class -- unlike the other five layer roots -- says nothing about it.
A System is DEFINED: somebody decides that MDM Product is a system worth tracking, and names it. Nobody harvests the list of systems that matter. Everything beneath a system is CAPTURED: databases, schemas, data structures, fields, keys and transformations are read from what is running.
Until v1.4 this class was itself declared a kind of Captured Object, which made every physical class captured by construction and put the five seeded Systems in breach of the harvest rules. The two axes are meant to be orthogonal; that shortcut collapsed them.""" .
pr:ConsumptionLayerObject a owl:Class ; rdfs:subClassOf pr:CapturedObject ;
rdfs:label "Consumption Layer Object" ; rdfs:label "Consumption Layer Object" ;
rdfs:comment "The BI estate. Separate from Physical because BI assets have their own lifecycle and their own owner: a dashboard belongs to its author, not to the Data Domain that owns the underlying data." . rdfs:comment "The BI estate. Separate from Physical because BI assets have their own lifecycle and their own owner: a dashboard belongs to its author, not to the Data Domain that owns the underlying data." ;
pr:hasShortLabel "Consumption" .
################################################################# #################################################################
# LAYER 1 - OWNERSHIP & CATEGORIZATION (curated) # LAYER 1 - OWNERSHIP & CATEGORIZATION (defined)
################################################################# #################################################################
pr:DataDomain a owl:Class ; rdfs:subClassOf pr:OwnershipLayerObject ; pr:DataDomain a owl:Class ; rdfs:subClassOf pr:OwnershipLayerObject ;
rdfs:label "Data Domain" . rdfs:label "Data Domain" ;
pr:hasShortLabel "Domain" .
pr:SubDomain a owl:Class ; rdfs:subClassOf pr:OwnershipLayerObject ; pr:SubDomain a owl:Class ; rdfs:subClassOf pr:OwnershipLayerObject ;
rdfs:label "Sub-Domain" . rdfs:label "Sub-Domain" ;
pr:hasShortLabel "Sub-domain" .
pr:BusinessObject a owl:Class ; rdfs:subClassOf pr:OwnershipLayerObject ; pr:BusinessObject a owl:Class ; rdfs:subClassOf pr:OwnershipLayerObject ;
rdfs:label "Business Object" ; rdfs:label "Business Object" ;
rdfs:comment "The single anchor of accountability, which is why it stays in the Ownership layer: it is the one object where a steward is declared, and Data Objects and Data Elements inherit from it. It carries NO business definition of its own -- its meaning comes from its aboutConcept. What it does carry is a perimeter: the subject it is about, and the notions needed to read it." . rdfs:comment "The single anchor of accountability, which is why it stays in the Ownership layer: it is the one object where a steward is declared, and Data Objects and Data Elements inherit from it. It carries NO business definition of its own -- its meaning comes from its aboutConcept. What it does carry is a perimeter: the subject it is about, and the notions needed to read it." ;
pr:hasShortLabel "Business object" .
################################################################# #################################################################
# LAYER 2 - BUSINESS (curated) # LAYER 2 - BUSINESS (defined)
################################################################# #################################################################
pr:BusinessConcept a owl:Class ; rdfs:subClassOf pr:BusinessLayerObject ; pr:BusinessConcept a owl:Class ; rdfs:subClassOf pr:BusinessLayerObject ;
rdfs:label "Business Concept" ; rdfs:label "Business Concept" ;
rdfs:comment "A shared business notion WITHOUT a formula. The presence of a formula is the sole discriminator against Metric (rule BR-003)." . rdfs:comment "A shared business notion WITHOUT a formula. The presence of a formula is the sole discriminator against Metric (rule BR-003)." ;
pr:hasShortLabel "Business concept" .
pr:Metric a owl:Class ; rdfs:subClassOf pr:BusinessLayerObject ; pr:Metric a owl:Class ; rdfs:subClassOf pr:BusinessLayerObject ;
rdfs:label "Metric" ; rdfs:label "Metric" ;
rdfs:comment "A quantifiable measurement WITH a formula, a unit and a grain. Under BR-013 a Metric carries the harmonised CALCULATION RULE; variants that differ only by analysis context are Data Elements linked back via computedBy." . rdfs:comment "A quantifiable measurement WITH a formula, a unit and a grain. Under BR-013 a Metric carries the harmonised CALCULATION RULE; variants that differ only by analysis context are Data Elements linked back via computedBy." ;
pr:hasShortLabel "Metric" .
pr:KPI a owl:Class ; rdfs:subClassOf pr:Metric ; pr:KPI a owl:Class ; rdfs:subClassOf pr:Metric ;
rdfs:label "Key Performance Indicator" ; rdfs:label "Key Performance Indicator" ;
rdfs:comment "An elevated Metric. Use this class rather than the deprecated isKPI flag." . rdfs:comment "An elevated Metric. Use this class rather than the deprecated isKPI flag." ;
pr:hasShortLabel "KPI" .
################################################################# #################################################################
# LAYER 3 - LOGICAL (curated) # LAYER 3 - LOGICAL (defined)
################################################################# #################################################################
pr:DataObject a owl:Class ; rdfs:subClassOf pr:LogicalLayerObject ; pr:DataObject a owl:Class ; rdfs:subClassOf pr:LogicalLayerObject ;
rdfs:label "Data Object" ; rdfs:label "Data Object" ;
rdfs:comment "The pivot: represents exactly one Business Object, is composed of Data Elements, and is materialized as one or more Physical Relations. The single junction between governance and technology." . rdfs:comment "The pivot: represents exactly one Business Object, is composed of Data Elements, and is materialized as one or more Physical Relations. The single junction between governance and technology." ;
pr:hasShortLabel "Data object" .
pr:DataElement a owl:Class ; rdfs:subClassOf pr:LogicalLayerObject ; pr:DataElement a owl:Class ; rdfs:subClassOf pr:LogicalLayerObject ;
rdfs:label "Data Element" ; rdfs:label "Data Element" ;
pr:curationMode "PROPOSED_CONFIRMED" ; pr:authoringMode "PROPOSED_CONFIRMED" ;
rdfs:comment "The finest curated unit. Its mapping to a physical Field is the ONLY place where the curated graph and the observed graph meet: proposed automatically by name and type matching, confirmed by a Data Steward." . rdfs:comment "The finest defined unit. Its mapping to a physical Field is the ONLY place where the defined graph and the captured graph meet: proposed automatically by name and type matching, confirmed by a Data Steward." ;
pr:hasShortLabel "Data element" .
pr:DataAsset a owl:Class ; rdfs:subClassOf pr:LogicalLayerObject ; pr:DataAsset a owl:Class ; rdfs:subClassOf pr:LogicalLayerObject ;
rdfs:label "Data Asset (deprecated)" ; rdfs:label "Data Asset (deprecated)" ;
@@ -282,66 +377,94 @@ pr:DataAsset a owl:Class ; rdfs:subClassOf pr:LogicalLayerObject ;
################################################################# #################################################################
# LAYER 4 - DELIVERY (curated) # LAYER 4 - DELIVERY (defined)
################################################################# #################################################################
pr:DataProduct a owl:Class ; rdfs:subClassOf pr:DeliveryLayerObject ; pr:DataProduct a owl:Class ; rdfs:subClassOf pr:DeliveryLayerObject ;
rdfs:label "Data Product" ; rdfs:label "Data Product" ;
rdfs:comment "The unit of delivery: what carries a version number, an owner, a changelog, a support channel and a list of consumers to warn. It has no single technical counterpart, and should not: physically it often lands on a Snowflake schema or a dbt project, but that is not what defines it. Operated by exactly one Sub-Domain -- it is a consumable, and there must be no ambiguity about who runs it. Candidate alignment: dcat:Dataset / DPROD." . rdfs:comment "The unit of delivery: what carries a version number, an owner, a changelog, a support channel and a list of consumers to warn. It has no single technical counterpart, and should not: physically it often lands on a Snowflake schema or a dbt project, but that is not what defines it. Operated by exactly one Sub-Domain -- it is a consumable, and there must be no ambiguity about who runs it. Candidate alignment: dcat:Dataset / DPROD." ;
pr:hasShortLabel "Data product" .
pr:DataInterface a owl:Class ; rdfs:subClassOf pr:DeliveryLayerObject ; pr:DataInterface a owl:Class ; rdfs:subClassOf pr:DeliveryLayerObject ;
rdfs:label "Data Interface" ; rdfs:label "Data Interface" ;
rdfs:comment "The channel through which a consumer actually gets the data: a Snowflake secure share or listing, a consumption schema with its grants, a file extract to a panel partner, an API or reverse-ETL feed. Not a Data Object (that is structure) and not a BI Data Source (that is one consumer among others). Verifiable: a grant, a share or an endpoint either exists or it does not. The output port in data-mesh terms." . rdfs:comment "The channel through which a consumer actually gets the data: a Snowflake secure share or listing, a consumption schema with its grants, a file extract to a panel partner, an API or reverse-ETL feed. Not a Data Object (that is structure) and not a BI Data Source (that is one consumer among others). Verifiable: a grant, a share or an endpoint either exists or it does not. The output port in data-mesh terms." ;
pr:hasShortLabel "Data interface" .
pr:DataContract a owl:Class ; rdfs:subClassOf pr:DeliveryLayerObject ; pr:DataContract a owl:Class ; rdfs:subClassOf pr:DeliveryLayerObject ;
rdfs:label "Data Contract" ; rdfs:label "Data Contract" ;
rdfs:comment "The service commitment attached to a Data Product: schema, freshness, quality thresholds, support, breaking-change notice period. Candidate alignment: ODCS (Bitol)." . rdfs:comment "The service commitment attached to a Data Product: schema, freshness, quality thresholds, support, breaking-change notice period. Candidate alignment: ODCS (Bitol)." ;
pr:hasShortLabel "Data contract" .
################################################################# #################################################################
# LAYER 5 - PHYSICAL (observed) # LAYER 5 - PHYSICAL (captured)
################################################################# #################################################################
pr:System a owl:Class ; rdfs:subClassOf pr:PhysicalLayerObject ; pr:System a owl:Class ; rdfs:subClassOf pr:PhysicalLayerObject , pr:DefinedObject ;
rdfs:label "System" ; rdfs:label "System" ;
pr:harvestSource "Manually seeded once, then referenced. Replaces the free-text hasSourceSystem." ; pr:authoringMode "ASSERTED" ;
rdfs:comment "A real system: Snowflake, MDM Product, MDM Customer, a panel provider feed, Tableau Server, Looker. Making this a node rather than a string is what makes 'what depends on MDM Product?' answerable." . rdfs:comment "A real system: Snowflake, MDM Product, MDM Customer, a panel provider feed, Tableau Server, Looker. Making this a node rather than a string is what makes 'what depends on MDM Product?' answerable." ;
pr:hasShortLabel "System" .
pr:Database a owl:Class ; rdfs:subClassOf pr:PhysicalLayerObject ; pr:Database a owl:Class ; rdfs:subClassOf pr:PhysicalLayerObject , pr:CapturedObject ;
rdfs:label "Database" ; rdfs:label "Database" ;
pr:harvestSource "SNOWFLAKE.INFORMATION_SCHEMA.DATABASES" . pr:harvestSource "SNOWFLAKE.INFORMATION_SCHEMA.DATABASES" ;
pr:hasShortLabel "Database" .
pr:Schema a owl:Class ; rdfs:subClassOf pr:PhysicalLayerObject ; pr:Schema a owl:Class ; rdfs:subClassOf pr:PhysicalLayerObject , pr:CapturedObject ;
rdfs:label "Schema" ; rdfs:label "Schema" ;
pr:harvestSource "SNOWFLAKE.INFORMATION_SCHEMA.SCHEMATA" . pr:harvestSource "SNOWFLAKE.INFORMATION_SCHEMA.SCHEMATA" ;
pr:hasShortLabel "Schema" .
pr:PhysicalRelation a owl:Class ; rdfs:subClassOf pr:PhysicalLayerObject ; pr:DataStructure a owl:Class ; rdfs:subClassOf pr:PhysicalLayerObject , pr:CapturedObject ;
rdfs:label "Physical Relation" ; rdfs:label "Data Structure" ;
pr:hasShortLabel "Table or view" ;
pr:harvestSource "SNOWFLAKE.INFORMATION_SCHEMA.TABLES" ; pr:harvestSource "SNOWFLAKE.INFORMATION_SCHEMA.TABLES" ;
rdfs:comment "Abstract parent of everything with columns. Specialised because impact traversal differs by kind: a View propagates impact to whatever it reads." . rdfs:comment """Anything that carries Fields and reads as a set of rows. Never instantiated directly: a harvest types the concrete kind, and a viewer shows that kind rather than this parent.
pr:BaseTable a owl:Class ; rdfs:subClassOf pr:PhysicalRelation ; WHY THE PARENT EXISTS. Three properties need exactly this set as their scope: storedIn from a Data Object, hasField and its inverse livesIn, and dependsOn. Without it each would have to be tripled, once per concrete kind, and the checks would go with them.
rdfs:label "Base Table" .
pr:View a owl:Class ; rdfs:subClassOf pr:PhysicalRelation ; WHY THE NAME CHANGED. It was PhysicalRelation, from Codd, where a relation IS a table. In an ontology "relation" reads as a property, and the word locks the class into the tabular world. A Neo4j node label, a Mongo collection and a Kafka topic all carry fields; none is a relation.
EXTENSION POINT for non-tabular storage. Nothing at Pernod Ricard is non-tabular today, so no such class is declared -- four terms were deprecated in v1.2 precisely because nobody instantiated them. The day a graph enters scope, the insertion is additive and touches no instance, because subClassOf is transitive:
pr:Dataset rdfs:subClassOf pr:DataStructure .
pr:Graph rdfs:subClassOf pr:DataStructure .
pr:BaseTable rdfs:subClassOf pr:Dataset . # instead of DataStructure
pr:inSchema rdfs:domain pr:Dataset . # instead of nothing
Existing BaseTable assertions stay DataStructure by entailment, every shape targeting DataStructure keeps working, and storedIn does not move. The extension point is paid once; the extension is paid when it arrives.""" .
pr:PhysicalRelation a owl:Class ; rdfs:subClassOf pr:PhysicalLayerObject , pr:CapturedObject ;
rdfs:label "Physical Relation (deprecated)" ;
owl:deprecated true ; dcterms:isReplacedBy pr:DataStructure .
pr:BaseTable a owl:Class ; rdfs:subClassOf pr:DataStructure ;
rdfs:label "Base Table" ;
pr:hasShortLabel "Base table" .
pr:View a owl:Class ; rdfs:subClassOf pr:DataStructure ;
rdfs:label "View" ; rdfs:label "View" ;
rdfs:comment "Depends on other relations through dependsOn, so impact is transitive." . rdfs:comment "Depends on other relations through dependsOn, so impact is transitive." ;
pr:hasShortLabel "View" .
pr:MaterializedView a owl:Class ; rdfs:subClassOf pr:PhysicalRelation ; pr:MaterializedView a owl:Class ; rdfs:subClassOf pr:DataStructure ;
rdfs:label "Materialized View (deprecated)" ; rdfs:label "Materialized View (deprecated)" ;
owl:deprecated true ; dcterms:isReplacedBy pr:View ; owl:deprecated true ; dcterms:isReplacedBy pr:View ;
rdfs:comment "A class standing in for an attribute: what distinguishes it is being derived, which dependsOn already states." . rdfs:comment "A class standing in for an attribute: what distinguishes it is being derived, which dependsOn already states." .
pr:ExternalTable a owl:Class ; rdfs:subClassOf pr:PhysicalRelation ; pr:ExternalTable a owl:Class ; rdfs:subClassOf pr:DataStructure ;
rdfs:label "External Table" ; rdfs:label "External Table" ;
rdfs:comment "Backed by files outside the warehouse. Marks the boundary where lineage leaves Snowflake." . rdfs:comment "Backed by files outside the warehouse. Marks the boundary where lineage leaves Snowflake." ;
pr:hasShortLabel "External table" .
pr:Field a owl:Class ; rdfs:subClassOf pr:PhysicalLayerObject ; pr:Field a owl:Class ; rdfs:subClassOf pr:PhysicalLayerObject , pr:CapturedObject ;
rdfs:label "Field" ; rdfs:label "Field" ;
pr:harvestSource "SNOWFLAKE.INFORMATION_SCHEMA.COLUMNS" ; pr:harvestSource "SNOWFLAKE.INFORMATION_SCHEMA.COLUMNS" ;
rdfs:comment "A physical column. The anchor of impact analysis: everything upstream reaches a Field, everything downstream starts from one." . rdfs:comment "A physical column. The anchor of impact analysis: everything upstream reaches a Field, everything downstream starts from one." ;
pr:hasShortLabel "Field" .
pr:KeyConstraint a owl:Class ; rdfs:subClassOf pr:PhysicalLayerObject ; pr:KeyConstraint a owl:Class ; rdfs:subClassOf pr:PhysicalLayerObject , pr:CapturedObject ;
rdfs:label "Key Constraint" ; rdfs:label "Key Constraint" ;
pr:harvestSource "SNOWFLAKE.INFORMATION_SCHEMA.TABLE_CONSTRAINTS + KEY_COLUMN_USAGE" ; pr:harvestSource "SNOWFLAKE.INFORMATION_SCHEMA.TABLE_CONSTRAINTS + KEY_COLUMN_USAGE" ;
rdfs:comment "Gives foreign keys a home. Rule OW-006 says an FK is not a Data Element - true, but it still needs somewhere to live, and it lets the asserted DataObject-references-DataObject edge be checked against reality." . rdfs:comment "Gives foreign keys a home. Rule OW-006 says an FK is not a Data Element - true, but it still needs somewhere to live, and it lets the asserted DataObject-references-DataObject edge be checked against reality." .
@@ -352,34 +475,39 @@ pr:PrimaryKey a owl:Class ; rdfs:subClassOf pr:KeyConstraint ;
pr:ForeignKeyConstraint a owl:Class ; rdfs:subClassOf pr:KeyConstraint ; pr:ForeignKeyConstraint a owl:Class ; rdfs:subClassOf pr:KeyConstraint ;
rdfs:label "Foreign Key Constraint" . rdfs:label "Foreign Key Constraint" .
pr:Transformation a owl:Class ; rdfs:subClassOf pr:PhysicalLayerObject , prov:Activity ; pr:Transformation a owl:Class ; rdfs:subClassOf pr:PhysicalLayerObject , pr:CapturedObject , prov:Activity ;
rdfs:label "Transformation" ; rdfs:label "Transformation" ;
pr:harvestSource "dbt manifest.json, Snowflake TASKS, Azure Data Factory pipelines" ; pr:harvestSource "dbt manifest.json, Snowflake TASKS, Azure Data Factory pipelines" ;
rdfs:comment "The code that turns inputs into outputs: a dbt model, a Snowflake task, an ADF pipeline. Subclass of prov:Activity so that lineage is expressed in a standard vocabulary rather than a private one." . rdfs:comment "The code that turns inputs into outputs: a dbt model, a Snowflake task, an ADF pipeline. Subclass of prov:Activity so that lineage is expressed in a standard vocabulary rather than a private one." ;
pr:hasShortLabel "Transformation" .
################################################################# #################################################################
# LAYER 6 - CONSUMPTION (observed) # LAYER 6 - CONSUMPTION (captured)
################################################################# #################################################################
pr:BIWorkspace a owl:Class ; rdfs:subClassOf pr:ConsumptionLayerObject ; pr:BIWorkspace a owl:Class ; rdfs:subClassOf pr:ConsumptionLayerObject ;
rdfs:label "BI Workspace" ; rdfs:label "BI Workspace" ;
pr:harvestSource "Tableau Metadata API (Project/Site) ; Looker API (Model)" . pr:harvestSource "Tableau Metadata API (Project/Site) ; Looker API (Model)" ;
pr:hasShortLabel "BI workspace" .
pr:BIDataSource a owl:Class ; rdfs:subClassOf pr:ConsumptionLayerObject ; pr:BIDataSource a owl:Class ; rdfs:subClassOf pr:ConsumptionLayerObject ;
rdfs:label "BI Data Source" ; rdfs:label "BI Data Source" ;
pr:harvestSource "Tableau Published Datasource ; Looker Explore" ; pr:harvestSource "Tableau Published Datasource ; Looker Explore" ;
rdfs:comment "The semantic surface a report is built on. This is where BI meets the warehouse." . rdfs:comment "The semantic surface a report is built on. This is where BI meets the warehouse." ;
pr:hasShortLabel "BI data source" .
pr:BIField a owl:Class ; rdfs:subClassOf pr:ConsumptionLayerObject ; pr:BIField a owl:Class ; rdfs:subClassOf pr:ConsumptionLayerObject ;
rdfs:label "BI Field" ; rdfs:label "BI Field" ;
pr:harvestSource "Tableau calculated fields ; LookML measures and dimensions" ; pr:harvestSource "Tableau calculated fields ; LookML measures and dimensions" ;
rdfs:comment "A field as a business user sees it. realizesMetric is what turns a certified Metric from a definition on paper into something measurable against the real BI estate." . rdfs:comment "A field as a business user sees it. exposesMetric is what turns a certified Metric from a definition on paper into something measurable against the real BI estate." ;
pr:hasShortLabel "BI field" .
pr:BIReport a owl:Class ; rdfs:subClassOf pr:ConsumptionLayerObject ; pr:BIReport a owl:Class ; rdfs:subClassOf pr:ConsumptionLayerObject ;
rdfs:label "BI Report" ; rdfs:label "BI Report" ;
pr:harvestSource "Tableau Workbook/Sheet ; Looker Look/Dashboard" ; pr:harvestSource "Tableau Workbook/Sheet ; Looker Look/Dashboard" ;
rdfs:comment "What a Data Governance Lead actually cares about in an impact analysis: not that a column changed, but which reports break and who to warn." . rdfs:comment "What a Data Governance Lead actually cares about in an impact analysis: not that a column changed, but which reports break and who to warn." ;
pr:hasShortLabel "BI report" .
################################################################# #################################################################
@@ -387,28 +515,34 @@ pr:BIReport a owl:Class ; rdfs:subClassOf pr:ConsumptionLayerObject ;
################################################################# #################################################################
pr:Actor a owl:Class ; rdfs:label "Actor" ; pr:Actor a owl:Class ; rdfs:label "Actor" ;
rdfs:comment "A person or a team. Deliberately NOT a MetaModelObject: people are not subject to ownership and lifecycle rules." . rdfs:comment "A person or a team. Deliberately NOT a MetaModelObject: people are not subject to ownership and lifecycle rules." ;
pr:hasShortLabel "Actor" .
pr:DataDomainOwner a owl:Class ; rdfs:subClassOf pr:Actor ; pr:DataDomainOwner a owl:Class ; rdfs:subClassOf pr:Actor ;
rdfs:label "Data Domain Owner" ; rdfs:label "Data Domain Owner" ;
rdfs:comment "Accountable for a Data Domain. Exactly one per domain." . rdfs:comment "Accountable for a Data Domain. Exactly one per domain." ;
pr:hasShortLabel "Domain owner" .
pr:SubDomainOwner a owl:Class ; rdfs:subClassOf pr:Actor ; pr:SubDomainOwner a owl:Class ; rdfs:subClassOf pr:Actor ;
rdfs:label "Sub-Domain Owner" ; rdfs:label "Sub-Domain Owner" ;
rdfs:comment "Accountable for a Sub-Domain. Exactly one per sub-domain." . rdfs:comment "Accountable for a Sub-Domain. Exactly one per sub-domain." ;
pr:hasShortLabel "Sub-domain owner" .
pr:DataGovernanceLead a owl:Class ; rdfs:subClassOf pr:Actor ; pr:DataGovernanceLead a owl:Class ; rdfs:subClassOf pr:Actor ;
rdfs:label "Data Governance Lead" ; rdfs:label "Data Governance Lead" ;
pr:hasAcronym "DGL" ; pr:hasAcronym "DGL" ;
rdfs:comment "Runs governance for a Data Domain. The acronym belongs in hasAcronym, not in the label (rule NR-005)." . rdfs:comment "Runs governance for a Data Domain. The acronym belongs in hasAcronym, not in the label (rule NR-005)." ;
pr:hasShortLabel "Data governance lead" .
pr:DataProductOwner a owl:Class ; rdfs:subClassOf pr:Actor ; pr:DataProductOwner a owl:Class ; rdfs:subClassOf pr:Actor ;
rdfs:label "Data Product Owner" ; rdfs:label "Data Product Owner" ;
rdfs:comment "Runs a Data Product day to day: releases, changelog, support, consumer communication. Belongs to the Sub-Domain that operates the product." . rdfs:comment "Runs a Data Product day to day: releases, changelog, support, consumer communication. Belongs to the Sub-Domain that operates the product." ;
pr:hasShortLabel "Product owner" .
pr:DataSteward a owl:Class ; rdfs:subClassOf pr:Actor ; pr:DataSteward a owl:Class ; rdfs:subClassOf pr:Actor ;
rdfs:label "Data Steward" ; rdfs:label "Data Steward" ;
rdfs:comment "Monitors a Business Object. Declared there and nowhere else: Data Objects and Data Elements inherit their steward, so that no column can ever have two answers to 'who watches this'." . rdfs:comment "Monitors a Business Object. Declared there and nowhere else: Data Objects and Data Elements inherit their steward, so that no column can ever have two answers to 'who watches this'." ;
pr:hasShortLabel "Steward" .
################################################################# #################################################################
@@ -525,10 +659,14 @@ pr:hasElement a owl:ObjectProperty ;
rdfs:label "has element" ; rdfs:label "has element" ;
rdfs:domain pr:DataObject ; rdfs:range pr:DataElement . rdfs:domain pr:DataObject ; rdfs:range pr:DataElement .
pr:references a owl:ObjectProperty ; pr:qualifiedBy a owl:ObjectProperty ;
rdfs:label "references" ; rdfs:label "qualified by" ;
rdfs:domain pr:DataObject ; rdfs:range pr:DataObject ; rdfs:domain pr:DataObject ; rdfs:range pr:DataObject ;
rdfs:comment "Foreign-key reference between Data Objects. NOT ownership. Now checkable against the harvested ForeignKeyConstraint objects." . rdfs:comment "The Data Objects that give this one its analysis context: a fact dataset is qualified by Product, Customer, Currency and Date. Physically a foreign key, which is why it is checkable against the harvested ForeignKeyConstraint objects -- but the name says the reason rather than the mechanism. NOT ownership. It is also what carries the grain of a fact object, whose own elements are measures and whose keys are not Data Elements (OW-006)." .
pr:references a owl:ObjectProperty ;
rdfs:label "references (deprecated)" ;
owl:deprecated true ; dcterms:isReplacedBy pr:qualifiedBy .
################################################################# #################################################################
# OBJECT PROPERTIES - Delivery # OBJECT PROPERTIES - Delivery
@@ -546,7 +684,7 @@ pr:exposes a owl:ObjectProperty ;
pr:servedBy a owl:ObjectProperty ; pr:servedBy a owl:ObjectProperty ;
rdfs:label "served by" ; rdfs:label "served by" ;
rdfs:domain pr:DataInterface ; rdfs:range pr:PhysicalRelation ; rdfs:domain pr:DataInterface ; rdfs:range pr:DataStructure ;
rdfs:comment "What physically backs the interface: the shared relations, the extract source." . rdfs:comment "What physically backs the interface: the shared relations, the extract source." .
pr:governedBy a owl:ObjectProperty , owl:FunctionalProperty ; pr:governedBy a owl:ObjectProperty , owl:FunctionalProperty ;
@@ -569,22 +707,39 @@ pr:usesDataObject a owl:ObjectProperty ;
################################################################# #################################################################
# OBJECT PROPERTIES - the curated / observed seam # OBJECT PROPERTIES - the defined / captured seam
# These two are the ONLY edges crossing from asserted to harvested. # These two are the ONLY edges crossing from asserted to harvested.
# Everything else is either pure governance or pure machine. # Everything else is either pure governance or pure machine.
################################################################# #################################################################
pr:materializedAs a owl:ObjectProperty ; pr:materializedAs a owl:ObjectProperty ;
rdfs:label "materialized as" ; rdfs:label "materialized as (deprecated)" ;
rdfs:domain pr:DataObject ; rdfs:range pr:PhysicalRelation ; owl:deprecated true ; dcterms:isReplacedBy pr:storedIn ;
pr:curationMode "PROPOSED_CONFIRMED" ; rdfs:comment "Redundant once storedIn serves two granularities. One verb fewer, and the seam counts one relation instead of two." .
rdfs:comment "Data Object to physical relation. Not functional: the same Data Object exists in DEV, UAT and PROD, and sometimes across several tables." .
pr:storedIn a owl:ObjectProperty ; pr:storedIn a owl:ObjectProperty ;
rdfs:label "stored in" ; rdfs:label "stored in" ;
pr:authoringMode "PROPOSED_CONFIRMED" ;
rdfs:comment """Where a logical artefact actually sits. One relation at two granularities, like represents:
DataObject storedIn DataStructure (a table, a view, later a node label)
DataElement storedIn Field (a column, later a node property)
Range left open and scoped by SHACL per class, because the two uses have different targets.
THE SEAM. This is the one relation crossing from what governance defines to what a harvester captures, and the one thing a steward confirms: the reconciliation job proposes the mapping on fully qualified name and type, a person accepts it. It is also what makes end-to-end impact analysis possible.
Not functional: the same object exists in DEV, UAT and PROD, and a Data Element may sit in several Fields -- see primaryLocation for the authoritative one. Renamed from physicalizedIn, which was jargon.""" .
pr:primaryLocation a owl:ObjectProperty , owl:FunctionalProperty ;
rdfs:subPropertyOf pr:storedIn ;
rdfs:label "primary location" ;
rdfs:domain pr:DataElement ; rdfs:range pr:Field ; rdfs:domain pr:DataElement ; rdfs:range pr:Field ;
pr:curationMode "PROPOSED_CONFIRMED" ; pr:authoringMode "PROPOSED_CONFIRMED" ;
rdfs:comment "Data Element to physical column, 1..n. Proposed by the reconciliation job on fullyQualifiedName and type, confirmed by a Data Steward. This single edge is what makes end-to-end impact analysis possible. Renamed from physicalizedIn, which was jargon: the relation says where the data sits." . rdfs:comment """The authoritative Field among the several a Data Element may sit in -- the others being copies, replicas or downstream materialisations.
A SUB-PROPERTY of storedIn on purpose, not a parallel relation. Entailment then guarantees the primary location is always among the locations, so the two can never disagree; two independent properties would allow a Field marked primary here and absent from storedIn there.
Primary and replica rather than master and slave: the industry has moved on, and it reads better to the business.""" .
pr:physicalizedIn a owl:ObjectProperty ; pr:physicalizedIn a owl:ObjectProperty ;
rdfs:label "physicalized in (deprecated)" ; rdfs:label "physicalized in (deprecated)" ;
@@ -598,7 +753,11 @@ pr:physicalizedIn a owl:ObjectProperty ;
pr:sourcedFrom a owl:ObjectProperty ; pr:sourcedFrom a owl:ObjectProperty ;
rdfs:label "sourced from" ; rdfs:label "sourced from" ;
rdfs:range pr:System ; rdfs:range pr:System ;
rdfs:comment "Replaces the free-text hasSourceSystem. Domain deliberately omitted: applies to Data Elements, Physical Relations and BI Data Sources alike." . rdfs:comment """The System holding the authoritative data for this object. Replaces the free-text hasSource, and answers a question that text could not: which objects depend on MDM Product.
Domain deliberately omitted -- it applies to Data Elements, Data Structures, Metrics and BI Data Sources alike, and the scope lives in the shapes.
It also settles a naming collision. As a string, "source" meant a system here and a BI surface in BIDataSource, two readings of one word. As a relation towards System, the two are different kinds of thing and the ambiguity disappears.""" .
pr:hasSystemType a owl:ObjectProperty , owl:FunctionalProperty ; pr:hasSystemType a owl:ObjectProperty , owl:FunctionalProperty ;
rdfs:label "has system type" ; rdfs:label "has system type" ;
@@ -614,21 +773,22 @@ pr:inDatabase a owl:ObjectProperty , owl:FunctionalProperty ;
pr:inSchema a owl:ObjectProperty , owl:FunctionalProperty ; pr:inSchema a owl:ObjectProperty , owl:FunctionalProperty ;
rdfs:label "in schema" ; rdfs:label "in schema" ;
rdfs:domain pr:PhysicalRelation ; rdfs:range pr:Schema . rdfs:range pr:Schema ;
rdfs:comment "Domain deliberately not declared, and the shapes must keep this OPTIONAL: Neo4j has no schema and BigQuery calls the level a dataset. A minCount 1 would break on the first harvest outside Snowflake. It is also the only property that would distinguish a tabular Dataset from a Graph, which is why that intermediate level is not declared yet -- one client does not pay for a level of hierarchy." .
pr:hasField a owl:ObjectProperty ; pr:hasField a owl:ObjectProperty ;
rdfs:label "has field" ; rdfs:label "has field" ;
rdfs:domain pr:PhysicalRelation ; rdfs:range pr:Field ; rdfs:domain pr:DataStructure ; rdfs:range pr:Field ;
owl:inverseOf pr:livesIn ; owl:inverseOf pr:livesIn ;
rdfs:comment "Domain widened from Table to PhysicalRelation so that views and external tables carry columns too. Inverse now declared - in v0.9 hasField and livesIn were inverses in fact but not in the model, so they were free to diverge." . rdfs:comment "Domain widened from Table to PhysicalRelation so that views and external tables carry columns too. Inverse now declared - in v0.9 hasField and livesIn were inverses in fact but not in the model, so they were free to diverge." .
pr:livesIn a owl:ObjectProperty , owl:FunctionalProperty ; pr:livesIn a owl:ObjectProperty , owl:FunctionalProperty ;
rdfs:label "lives in" ; rdfs:label "lives in" ;
rdfs:domain pr:Field ; rdfs:range pr:PhysicalRelation . rdfs:domain pr:Field ; rdfs:range pr:DataStructure .
pr:deployedIn a owl:ObjectProperty , owl:FunctionalProperty ; pr:deployedIn a owl:ObjectProperty , owl:FunctionalProperty ;
rdfs:label "deployed in" ; rdfs:label "deployed in" ;
rdfs:domain pr:ObservedObject ; rdfs:range pr:Environment . rdfs:domain pr:CapturedObject ; rdfs:range pr:Environment .
pr:constrainsRelation a owl:ObjectProperty , owl:FunctionalProperty ; pr:constrainsRelation a owl:ObjectProperty , owl:FunctionalProperty ;
rdfs:label "constrains relation (deprecated)" ; rdfs:label "constrains relation (deprecated)" ;
@@ -651,15 +811,15 @@ pr:toField a owl:ObjectProperty ;
pr:consumes a owl:ObjectProperty ; pr:consumes a owl:ObjectProperty ;
rdfs:subPropertyOf prov:used ; rdfs:subPropertyOf prov:used ;
rdfs:label "consumes" ; rdfs:label "consumes" ;
rdfs:domain pr:Transformation ; rdfs:range pr:PhysicalRelation . rdfs:domain pr:Transformation ; rdfs:range pr:DataStructure .
pr:produces a owl:ObjectProperty ; pr:produces a owl:ObjectProperty ;
rdfs:label "produces" ; rdfs:label "produces" ;
rdfs:domain pr:Transformation ; rdfs:range pr:PhysicalRelation . rdfs:domain pr:Transformation ; rdfs:range pr:DataStructure .
pr:dependsOn a owl:ObjectProperty , owl:TransitiveProperty ; pr:dependsOn a owl:ObjectProperty , owl:TransitiveProperty ;
rdfs:label "depends on" ; rdfs:label "depends on" ;
rdfs:domain pr:PhysicalRelation ; rdfs:range pr:PhysicalRelation ; rdfs:domain pr:DataStructure ; rdfs:range pr:DataStructure ;
pr:harvestSource "SNOWFLAKE.ACCOUNT_USAGE.OBJECT_DEPENDENCIES" ; pr:harvestSource "SNOWFLAKE.ACCOUNT_USAGE.OBJECT_DEPENDENCIES" ;
rdfs:comment "Relation-level dependency, view to table. Transitive by nature; in practice compute the closure at query time in Snowflake rather than materialising it with a reasoner." . rdfs:comment "Relation-level dependency, view to table. Transitive by nature; in practice compute the closure at query time in Snowflake rather than materialising it with a reasoner." .
@@ -681,7 +841,7 @@ pr:inWorkspace a owl:ObjectProperty , owl:FunctionalProperty ;
pr:biConsumes a owl:ObjectProperty ; pr:biConsumes a owl:ObjectProperty ;
rdfs:label "BI consumes" ; rdfs:label "BI consumes" ;
rdfs:domain pr:BIDataSource ; rdfs:range pr:PhysicalRelation ; rdfs:domain pr:BIDataSource ; rdfs:range pr:DataStructure ;
pr:harvestSource "Tableau Metadata API (GraphQL) ; Looker Explore SQL" . pr:harvestSource "Tableau Metadata API (GraphQL) ; Looker Explore SQL" .
pr:hasBIField a owl:ObjectProperty ; pr:hasBIField a owl:ObjectProperty ;
@@ -699,11 +859,11 @@ pr:biDerivedFrom a owl:ObjectProperty ;
rdfs:domain pr:BIField ; rdfs:range pr:Field ; rdfs:domain pr:BIField ; rdfs:range pr:Field ;
rdfs:comment "Closes the last gap in the impact chain: from a dashboard field back to the warehouse column." . rdfs:comment "Closes the last gap in the impact chain: from a dashboard field back to the warehouse column." .
pr:realizesMetric a owl:ObjectProperty , owl:FunctionalProperty ; pr:exposesMetric a owl:ObjectProperty , owl:FunctionalProperty ;
rdfs:label "realizes metric" ; rdfs:label "exposes metric" ;
rdfs:domain pr:BIField ; rdfs:range pr:Metric ; rdfs:domain pr:BIField ; rdfs:range pr:Metric ;
pr:curationMode "PROPOSED_CONFIRMED" ; pr:authoringMode "PROPOSED_CONFIRMED" ;
rdfs:comment """The harmonisation measure: how many distinct BI Fields realize one certified Metric, and do their expressions agree. Turns 'we harmonised the definitions' from a claim into a count over the real BI estate. rdfs:comment """The harmonisation measure: how many distinct BI Fields expose one certified Metric, and do their expressions agree. Turns 'we harmonised the definitions' from a claim into a count over the real BI estate.
In principle this is derivable from the lineage chain Metric -> DataElement -> Field -> BIField, since lineage carries semantic inheritance. In practice that chain will have holes -- four hops, one of them PROPOSED_CONFIRMED, another harvested from an API that does not resolve custom SQL, extracts or cascaded calculated fields -- and the question it answers is too important to depend on them. In principle this is derivable from the lineage chain Metric -> DataElement -> Field -> BIField, since lineage carries semantic inheritance. In practice that chain will have holes -- four hops, one of them PROPOSED_CONFIRMED, another harvested from an API that does not resolve custom SQL, extracts or cascaded calculated fields -- and the question it answers is too important to depend on them.
@@ -730,7 +890,7 @@ pr:ownedBy a owl:ObjectProperty ;
pr:owningDomain a owl:ObjectProperty , owl:FunctionalProperty ; pr:owningDomain a owl:ObjectProperty , owl:FunctionalProperty ;
rdfs:label "owning Data Domain" ; rdfs:range pr:DataDomain ; rdfs:label "owning Data Domain" ; rdfs:range pr:DataDomain ;
rdfs:comment """The single Data Domain with authority to define the object's rule. Absent on Observed Objects: a Snowflake column is attributed through the Data Element mapped to it, never asserted directly. rdfs:comment """The single Data Domain with authority to define the object's rule. Absent on Captured Objects: a Snowflake column is attributed through the Data Element mapped to it, never asserted directly.
Unlike the steward, this one is RE-DECLARABLE on a Data Element. It is inherited by default from the parent chain, but OW-002 attributes an element to the domain that defines its RULE, and a borrowed element -- Brand Code sitting in a Sell Out fact object -- has its rule elsewhere. A Data Element whose owningDomain differs from its parent's is a borrowed element: a signal to check against its represents, not an anomaly to fix.""" . Unlike the steward, this one is RE-DECLARABLE on a Data Element. It is inherited by default from the parent chain, but OW-002 attributes an element to the domain that defines its RULE, and a borrowed element -- Brand Code sitting in a Sell Out fact object -- has its rule elsewhere. A Data Element whose owningDomain differs from its parent's is a borrowed element: a signal to check against its represents, not an anomaly to fix.""" .
@@ -769,11 +929,18 @@ pr:hasActivationStatus a owl:ObjectProperty , owl:FunctionalProperty ;
pr:hasIdentifier a owl:DatatypeProperty , owl:FunctionalProperty ; rdfs:label "identifier" ; rdfs:range xsd:string . pr:hasIdentifier a owl:DatatypeProperty , owl:FunctionalProperty ; rdfs:label "identifier" ; rdfs:range xsd:string .
pr:hasName a owl:DatatypeProperty , owl:FunctionalProperty ; rdfs:label "canonical name" ; rdfs:range xsd:string . pr:hasName a owl:DatatypeProperty , owl:FunctionalProperty ; rdfs:label "canonical name" ; rdfs:range xsd:string .
pr:hasShortLabel a owl:DatatypeProperty , owl:FunctionalProperty ;
rdfs:label "short label" ; rdfs:range xsd:string ;
rdfs:comment """The name screens use, where the canonical name is the exhaustive one: Domain for Data Domain, SODH Gold for Sell Out Data Hub - Gold.
Declared rather than left to each interface. The same object was being called two different things depending on where you looked, because every viewer kept a display dictionary of its own. One name is authoritative, the other is short, and both live in the model.""" .
pr:hasStatus a owl:DatatypeProperty ; rdfs:label "status" ; rdfs:range xsd:string . pr:hasStatus a owl:DatatypeProperty ; rdfs:label "status" ; rdfs:range xsd:string .
pr:hasVersion a owl:DatatypeProperty ; rdfs:label "version" ; rdfs:range xsd:string . pr:hasVersion a owl:DatatypeProperty ; rdfs:label "version" ; rdfs:range xsd:string .
pr:createdOn a owl:DatatypeProperty ; rdfs:label "created on" ; rdfs:range xsd:date . pr:createdOn a owl:DatatypeProperty ; rdfs:label "created on" ; rdfs:range xsd:date .
pr:lastReviewedOn a owl:DatatypeProperty ; rdfs:label "last reviewed on" ; rdfs:range xsd:date . pr:lastReviewedOn a owl:DatatypeProperty ; rdfs:label "last reviewed on" ; rdfs:range xsd:date .
pr:hasSource a owl:DatatypeProperty ; rdfs:label "source of truth" ; rdfs:range xsd:string ; pr:hasSource a owl:DatatypeProperty ; rdfs:label "source of truth (deprecated)" ; rdfs:range xsd:string ;
owl:deprecated true ; dcterms:isReplacedBy pr:sourcedFrom ;
rdfs:comment """The system holding the authoritative data for this object: SODH Gold, MDM Product, Group Finance. rdfs:comment """The system holding the authoritative data for this object: SODH Gold, MDM Product, Group Finance.
ONE MEANING ONLY. It had drifted into naming the document a record came from ("SODH back-doc v0.7"), and into packing a system and a relation into one string ("SODH Gold / F_SELL_OUT"). The relation belongs in physicalName; the document belongs nowhere -- provenance of a record is not a property of the thing it describes. ONE MEANING ONLY. It had drifted into naming the document a record came from ("SODH back-doc v0.7"), and into packing a system and a relation into one string ("SODH Gold / F_SELL_OUT"). The relation belongs in physicalName; the document belongs nowhere -- provenance of a record is not a property of the thing it describes.
@@ -834,7 +1001,7 @@ pr:sourceIdentifier a owl:DatatypeProperty , owl:FunctionalProperty ;
rdfs:comment "Native immutable id in the source system (Snowflake object id, Tableau LUID, Looker id). Makes reconciliation idempotent and survives renames, which names do not." . rdfs:comment "Native immutable id in the source system (Snowflake object id, Tableau LUID, Looker id). Makes reconciliation idempotent and survives renames, which names do not." .
pr:harvestedOn a owl:DatatypeProperty ; rdfs:label "harvested on" ; rdfs:range xsd:dateTime ; pr:harvestedOn a owl:DatatypeProperty ; rdfs:label "harvested on" ; rdfs:range xsd:dateTime ;
rdfs:comment "Staleness marker. An Observed Object not refreshed by the last run is a deletion candidate, not a fact." . rdfs:comment "Staleness marker. A Captured Object not refreshed by the last run is a deletion candidate, not a fact." .
pr:hasPhysicalDataType a owl:DatatypeProperty ; rdfs:label "physical data type" ; rdfs:range xsd:string . pr:hasPhysicalDataType a owl:DatatypeProperty ; rdfs:label "physical data type" ; rdfs:range xsd:string .
pr:isNullable a owl:DatatypeProperty ; rdfs:label "is nullable" ; rdfs:range xsd:boolean . pr:isNullable a owl:DatatypeProperty ; rdfs:label "is nullable" ; rdfs:range xsd:boolean .
@@ -863,7 +1030,7 @@ pr:hasExpression a owl:DatatypeProperty ; rdfs:label "BI expression" ; rdfs:rang
# DISJOINTNESS # DISJOINTNESS
################################################################# #################################################################
[] a owl:AllDisjointClasses ; owl:members ( pr:CuratedObject pr:ObservedObject ) . [] a owl:AllDisjointClasses ; owl:members ( pr:DefinedObject pr:CapturedObject ) .
[] a owl:AllDisjointClasses ; owl:members ( [] a owl:AllDisjointClasses ; owl:members (
pr:OwnershipLayerObject pr:BusinessLayerObject pr:LogicalLayerObject pr:OwnershipLayerObject pr:BusinessLayerObject pr:LogicalLayerObject
@@ -875,7 +1042,7 @@ pr:hasExpression a owl:DatatypeProperty ; rdfs:label "BI expression" ; rdfs:rang
[] a owl:AllDisjointClasses ; owl:members ( pr:BusinessConcept pr:Metric ) . [] a owl:AllDisjointClasses ; owl:members ( pr:BusinessConcept pr:Metric ) .
[] a owl:AllDisjointClasses ; owl:members ( [] a owl:AllDisjointClasses ; owl:members (
pr:BaseTable pr:View pr:MaterializedView pr:ExternalTable ) . pr:BaseTable pr:View pr:ExternalTable ) .
[] a owl:AllDisjointClasses ; owl:members ( pr:PrimaryKey pr:ForeignKeyConstraint ) . [] a owl:AllDisjointClasses ; owl:members ( pr:PrimaryKey pr:ForeignKeyConstraint ) .
+64 -31
View File
@@ -35,13 +35,13 @@ DOMAIN_STYLE = OrderedDict([
]) ])
MULTI = {"composedOf", "usesConcept", "hasElement", "hasMetric", "measures", MULTI = {"composedOf", "usesConcept", "hasElement", "hasMetric", "measures",
"computedBy", "references", "hasGrainElement", "monitoredBy", "hasSynonym", "computedBy", "qualifiedBy", "hasGrainElement", "monitoredBy", "hasSynonym",
"hasUnit", "hasSource", "packages", "exposes"} "hasUnit", "hasSource", "packages", "exposes"}
SUBJECT_DOMAIN = "DD_06" # the hub owner: everything else is a dependency SUBJECT_DOMAIN = "DD_06" # the hub owner: everything else is a dependency
# Every user-facing label in one place. Hyphenated throughout, and never the # Every user-facing label in one place. Hyphenated throughout, and never the
# raw class name: "Physical Relation" and "Physicalized in" told a business # raw class name: "Data Structure" and "Physicalized in" told a business
# reader nothing. # reader nothing.
LABELS = { LABELS = {
"DataDomain": "Domain", "SubDomain": "Sub-domain", "DataDomain": "Domain", "SubDomain": "Sub-domain",
@@ -49,14 +49,17 @@ LABELS = {
"Metric": "Metric", "DataObject": "Data object", "DataElement": "Data element", "Metric": "Metric", "DataObject": "Data object", "DataElement": "Data element",
"BaseTable": "Base table", "Field": "Field", "BaseTable": "Base table", "Field": "Field",
"DataProduct": "Data product", "DataContract": "Data contract", "DataProduct": "Data product", "DataContract": "Data contract",
"System": "System",
"DataInterface": "Data interface", "DataInterface": "Data interface",
} }
# Which ownership roles a card shows. The Steward section is gone: one section, # Which ownership roles a card shows. The Steward section is gone: one section,
# different roles per object, so the reader always looks in the same place. # different roles per object, so the reader always looks in the same place.
ROLES = { ROLES = {
"DataDomain": [("Domain owner", "owner"), ("Data-governance lead", "dgl")], # role labels mirror the short labels the T-Box carries on the actor
"SubDomain": [("Sub-domain owner", "subOwner"), ("Data-governance lead", "dgl_up")], # classes, so the two viewers name the same role the same way
"DataDomain": [("Domain owner", "owner"), ("Data governance lead", "dgl")],
"SubDomain": [("Sub-domain owner", "subOwner"), ("Data governance lead", "dgl_up")],
"BusinessObject": [("Steward", "steward")], "BusinessObject": [("Steward", "steward")],
"BusinessConcept": [("Steward", "steward")], "BusinessConcept": [("Steward", "steward")],
"Metric": [("Steward", "steward")], "Metric": [("Steward", "steward")],
@@ -180,7 +183,10 @@ def build(subjects):
nodes[subj] = { nodes[subj] = {
"id": subj, "id": subj,
"kind": t, "kind": t,
# the model carries both names now: hasShortLabel for screens,
# hasName for the record. The viewer stops keeping its own list.
"label": LABELS.get(t, t), "label": LABELS.get(t, t),
"shortName": rec.get("hasShortLabel", ""),
"ident": rec.get("hasIdentifier", ""), "ident": rec.get("hasIdentifier", ""),
"name": rec.get("hasName", subj.replace("ex:", "")), "name": rec.get("hasName", subj.replace("ex:", "")),
"domain": domain_of(subj, rec, subjects), "domain": domain_of(subj, rec, subjects),
@@ -193,6 +199,8 @@ def build(subjects):
"physical": rec.get("physicalName", ""), "physical": rec.get("physicalName", ""),
"format": rec.get("hasFormat", ""), "format": rec.get("hasFormat", ""),
"source": ", ".join(as_list(rec.get("hasSource"))), "source": ", ".join(as_list(rec.get("hasSource"))),
"sourcedFrom": rec.get("sourcedFrom", ""),
"fqn": "",
"version": rec.get("hasVersion", ""), "version": rec.get("hasVersion", ""),
"operatedBy": rec.get("operatedBy", ""), "operatedBy": rec.get("operatedBy", ""),
"governedBy": rec.get("governedBy", ""), "governedBy": rec.get("governedBy", ""),
@@ -208,7 +216,7 @@ def build(subjects):
"computedBy": as_list(rec.get("computedBy")), "computedBy": as_list(rec.get("computedBy")),
"hasElement": as_list(rec.get("hasElement")), "hasElement": as_list(rec.get("hasElement")),
"grain": as_list(rec.get("hasGrainElement")), "grain": as_list(rec.get("hasGrainElement")),
"references": as_list(rec.get("references")), "qualifiedBy": as_list(rec.get("qualifiedBy")),
"steward": as_list(rec.get("monitoredBy")) or as_list(rec.get("ownedBy")), "steward": as_list(rec.get("monitoredBy")) or as_list(rec.get("ownedBy")),
"dgl": rec.get("hasDGL", ""), "dgl": rec.get("hasDGL", ""),
"owner": rec.get("hasDomainOwner", ""), "owner": rec.get("hasDomainOwner", ""),
@@ -239,6 +247,7 @@ def build(subjects):
phys.setdefault(pid, {"id": pid, "kind": "BaseTable", phys.setdefault(pid, {"id": pid, "kind": "BaseTable",
"label": LABELS["BaseTable"], "label": LABELS["BaseTable"],
"name": n["physical"], "domain": n["domain"], "name": n["physical"], "domain": n["domain"],
"fqn": n["physical"],
"derived": True, "backs": []}) "derived": True, "backs": []})
phys[pid]["backs"].append(n["id"]) phys[pid]["backs"].append(n["id"])
n["materializedAs"] = pid n["materializedAs"] = pid
@@ -251,6 +260,7 @@ def build(subjects):
phys.setdefault(pid, {"id": pid, "kind": "Field", phys.setdefault(pid, {"id": pid, "kind": "Field",
"label": LABELS["Field"], "name": n["physical"], "label": LABELS["Field"], "name": n["physical"],
"domain": n["domain"], "derived": True, "domain": n["domain"], "derived": True,
"fqn": (rel + "." if rel else "") + n["physical"],
"relation": "phys:" + rel if rel else "", "relation": "phys:" + rel if rel else "",
"backs": []}) "backs": []})
phys[pid]["backs"].append(n["id"]) phys[pid]["backs"].append(n["id"])
@@ -258,13 +268,14 @@ def build(subjects):
for p in phys.values(): for p in phys.values():
p.setdefault("ident", ""); p.setdefault("arb", False) p.setdefault("ident", ""); p.setdefault("arb", False)
p.setdefault("label", LABELS.get(p["kind"], p["kind"])) p.setdefault("label", LABELS.get(p["kind"], p["kind"]))
p.setdefault("shortName", "")
for k in ("definition","rule","formula","physical","format","source","belongsTo", for k in ("definition","rule","formula","physical","format","source","belongsTo",
"about","represents","status","dgl","owner","subOwner","activation", "about","represents","status","dgl","owner","subOwner","activation",
"version","operatedBy","governedBy","productOwner","storedIn", "version","operatedBy","governedBy","productOwner","storedIn",
"materializedAs","relation","stewardFrom"): "materializedAs","relation","stewardFrom","fqn","sourcedFrom"):
p.setdefault(k, "") p.setdefault(k, "")
for k in ("uses","measures","hasMetric","computedBy","hasElement","grain", for k in ("uses","measures","hasMetric","computedBy","hasElement","grain",
"references","steward","units","computedFrom","packages","exposes", "qualifiedBy","steward","units","computedFrom","packages","exposes",
"steward_inherited"): "steward_inherited"):
p.setdefault(k, []) p.setdefault(k, [])
p.setdefault("route", "") p.setdefault("route", "")
@@ -777,7 +788,7 @@ var G = {mode:"strata", doms:[], sel:null, hot:null, labels:true,
show:{DataDomain:true, SubDomain:true, BusinessObject:true, show:{DataDomain:true, SubDomain:true, BusinessObject:true,
BusinessConcept:true, Metric:true, DataObject:true, BusinessConcept:true, Metric:true, DataObject:true,
DataElement:false, DataProduct:true, DataContract:true, DataElement:false, DataProduct:true, DataContract:true,
DataInterface:true, BaseTable:true, Field:false}, DataInterface:true, System:true, BaseTable:true, Field:false},
view:{x:0,y:0,s:1}, nodes:[], edges:[]}; view:{x:0,y:0,s:1}, nodes:[], edges:[]};
var NW=126, NH=32, LANEPAD=26, ROWGAP=15, BANDGAP=34, GUT=142; var NW=126, NH=32, LANEPAD=26, ROWGAP=15, BANDGAP=34, GUT=142;
@@ -787,12 +798,12 @@ var ROW = {DataDomain:0, SubDomain:1, BusinessObject:2,
BusinessConcept:0, Metric:1, BusinessConcept:0, Metric:1,
DataObject:0, DataElement:1, DataObject:0, DataElement:1,
DataProduct:0, DataContract:1, DataInterface:1, DataProduct:0, DataContract:1, DataInterface:1,
BaseTable:0, Field:1}; System:0, BaseTable:1, Field:2};
var LAYER = {DataDomain:0, SubDomain:0, BusinessObject:0, var LAYER = {DataDomain:0, SubDomain:0, BusinessObject:0,
BusinessConcept:1, Metric:1, BusinessConcept:1, Metric:1,
DataObject:2, DataElement:2, DataObject:2, DataElement:2,
DataProduct:3, DataInterface:3, DataContract:3, DataProduct:3, DataInterface:3, DataContract:3,
BaseTable:4, Field:4}; System:4, BaseTable:4, Field:4};
var BANDS = [ var BANDS = [
{t:"Ownership & Categorization", g:"who is accountable"}, {t:"Ownership & Categorization", g:"who is accountable"},
{t:"Business", g:"what it means"}, {t:"Business", g:"what it means"},
@@ -805,7 +816,7 @@ var KINDS = [["DataDomain","Domains"],["SubDomain","Sub-domains"],
["Metric","Metrics"],["DataObject","Data objects"], ["Metric","Metrics"],["DataObject","Data objects"],
["DataElement","Data elements"],["DataProduct","Data products"], ["DataElement","Data elements"],["DataProduct","Data products"],
["DataContract","Data contracts"],["DataInterface","Data interfaces"], ["DataContract","Data contracts"],["DataInterface","Data interfaces"],
["BaseTable","Base tables"],["Field","Fields"]]; ["System","Systems"],["BaseTable","Base tables"],["Field","Fields"]];
function subjectColour(dom){ return dom === M.subject ? "#1E4B9B" : "#8A6A28"; } function subjectColour(dom){ return dom === M.subject ? "#1E4B9B" : "#8A6A28"; }
@@ -831,8 +842,8 @@ function buildGraph(){
n.measures.forEach(function(c){ edge(n.id, c, "measures"); }); n.measures.forEach(function(c){ edge(n.id, c, "measures"); });
n.computedBy.forEach(function(d){ edge(n.id, d, "computed by"); }); n.computedBy.forEach(function(d){ edge(n.id, d, "computed by"); });
n.hasElement.forEach(function(d){ edge(n.id, d, "has element"); }); n.hasElement.forEach(function(d){ edge(n.id, d, "has element"); });
n.references.forEach(function(d){ edge(n.id, d, "references"); }); n.qualifiedBy.forEach(function(d){ edge(n.id, d, "qualified by"); });
if(n.materializedAs) edge(n.id, n.materializedAs, "materialized as"); if(n.sourcedFrom) edge(n.id, n.sourcedFrom, "sourced from");
if(n.storedIn) edge(n.id, n.storedIn, "stored in"); if(n.storedIn) edge(n.id, n.storedIn, "stored in");
if(n.operatedBy) edge(n.id, n.operatedBy, "operated by"); if(n.operatedBy) edge(n.id, n.operatedBy, "operated by");
if(n.governedBy) edge(n.id, n.governedBy, "governed by"); if(n.governedBy) edge(n.id, n.governedBy, "governed by");
@@ -1113,10 +1124,12 @@ function renderGraphShell(){
}).join("") }).join("")
+ '<div class="gsec">Options</div>' + '<div class="gsec">Options</div>'
+ '<label><input type="checkbox" id="g-lab" checked> Relation names</label>' + '<label><input type="checkbox" id="g-lab" checked> Relation names</label>'
+ '<select id="g-dom"><option value="">All domains</option>' + '<div class="gsec">Domains</div>'
+ M.domains.map(function(d){ return '<option value="'+d.code+'"' + M.domains.map(function(d){
+ (G.dom===d.code?" selected":"") + '>'+esc(d.label)+'</option>'; }).join("") return '<label><input type="checkbox" data-dm="' + d.code + '"'
+ '</select>' + (!G.doms.length || G.doms.indexOf(d.code) > -1 ? " checked" : "")
+ '> <i style="background:' + (d.code === M.subject ? "#1E4B9B" : "#8A6A28")
+ '"></i>' + esc(d.label) + '</label>'; }).join("")
+ '<div class="mono" id="g-count"></div></div>'); + '<div class="mono" id="g-count"></div></div>');
h.push('<div class="gzoom"><button id="g-in">+</button>' h.push('<div class="gzoom"><button id="g-in">+</button>'
+ '<button id="g-out">\u2212</button><button id="g-fit" style="font-size:12px">' + '<button id="g-out">\u2212</button><button id="g-fit" style="font-size:12px">'
@@ -1222,7 +1235,7 @@ var HIST = [];
/* reverse indexes, built once */ /* reverse indexes, built once */
var PARENT_DO = {}, SUBS_OF = {}, BOS_OF = {}, BACKS = {}, var PARENT_DO = {}, SUBS_OF = {}, BOS_OF = {}, BACKS = {},
METRICS_OF = {}, REALISED_BY = {}; METRICS_OF = {}, REALISED_BY = {}, ABOUT_OF = {}, USES_OF = {};
M.nodes.forEach(function(n){ M.nodes.forEach(function(n){
n.hasElement.forEach(function(de){ PARENT_DO[de] = n.id; }); n.hasElement.forEach(function(de){ PARENT_DO[de] = n.id; });
if(n.kind === "SubDomain" && n.belongsTo) (SUBS_OF[n.belongsTo] = SUBS_OF[n.belongsTo]||[]).push(n.id); if(n.kind === "SubDomain" && n.belongsTo) (SUBS_OF[n.belongsTo] = SUBS_OF[n.belongsTo]||[]).push(n.id);
@@ -1230,6 +1243,8 @@ M.nodes.forEach(function(n){
(n.backs||[]).forEach(function(b){ (BACKS[n.id] = BACKS[n.id]||[]).push(b); }); (n.backs||[]).forEach(function(b){ (BACKS[n.id] = BACKS[n.id]||[]).push(b); });
/* a concept needs to show the metrics that measure it, not only the reverse */ /* a concept needs to show the metrics that measure it, not only the reverse */
n.measures.forEach(function(bc){ (METRICS_OF[bc] = METRICS_OF[bc]||[]).push(n.id); }); n.measures.forEach(function(bc){ (METRICS_OF[bc] = METRICS_OF[bc]||[]).push(n.id); });
if(n.about) (ABOUT_OF[n.about] = ABOUT_OF[n.about]||[]).push(n.id);
n.uses.forEach(function(u){ (USES_OF[u] = USES_OF[u]||[]).push(n.id); });
if(n.kind === "DataObject" && n.represents) if(n.kind === "DataObject" && n.represents)
(REALISED_BY[n.represents] = REALISED_BY[n.represents]||[]).push(n.id); (REALISED_BY[n.represents] = REALISED_BY[n.represents]||[]).push(n.id);
}); });
@@ -1256,8 +1271,15 @@ function paint(id){
+ esc(byId[HIST[HIST.length-2]].name) + '</button>'); + esc(byId[HIST[HIST.length-2]].name) + '</button>');
h.push('<div class="kind" style="color:' + subjectColour(n.domain) + '">' h.push('<div class="kind" style="color:' + subjectColour(n.domain) + '">'
+ esc(n.label || n.kind) + '</div>'); + esc(n.label || n.kind) + '</div>');
h.push('<h3>' + esc(n.name) + '</h3>'); h.push('<h3>' + esc(n.shortName || n.name) + '</h3>');
h.push('<div class="iri">' + esc(n.ident || n.id) + '</div>'); if(n.shortName && n.shortName !== n.name)
h.push('<div class="row" style="border:0;padding:0 0 8px"><span class="mono">'
+ esc(n.name) + '</span><span></span></div>');
/* a derived physical node has no governance identifier -- showing its
internal id as a subtitle read like one. The qualified name is an
attribute, and belongs in the attributes block with everything else. */
if(n.ident) h.push('<div class="iri">' + esc(n.ident) + '</div>');
else h.push('<div style="height:8px"></div>');
if(n.arb) h.push('<div class="note"><b>Awaiting arbitration.</b> Proposed by the ' if(n.arb) h.push('<div class="note"><b>Awaiting arbitration.</b> Proposed by the '
+ 'Data Governance Office; ' + esc(domLabel(n.domain)) + 'Data Governance Office; ' + esc(domLabel(n.domain))
+ ' owns this perimeter and has not ratified it. Cannot be published (OW-007).</div>'); + ' owns this perimeter and has not ratified it. Cannot be published (OW-007).</div>');
@@ -1270,15 +1292,20 @@ function paint(id){
var rows = []; var rows = [];
function at(label, v){ if(v) rows.push([label, v]); } function at(label, v){ if(v) rows.push([label, v]); }
at("Status", n.status); at("Status", n.status);
at("Source-of-truth", n.source); if(n.kind !== "DataProduct") at("Source-of-truth", n.source);
if(n.kind === "DataElement"){ if(n.kind === "DataElement"){
at("Base table", byId[n.storedIn] && byId[byId[n.storedIn].relation]
? byId[byId[n.storedIn].relation].name : "");
at("Table field", n.physical); at("Table field", n.physical);
at("Unit", n.units.join(", ")); at("Unit", n.units.join(", "));
at("Format", n.format); at("Format", n.format);
} else if(n.kind === "DataObject"){
at("Base table", n.physical);
} else if(n.kind === "BaseTable" || n.kind === "Field"){
at("Fully qualified name", n.fqn);
} else { } else {
at("Unit" + (n.units.length > 1 ? "s" : ""), n.units.join(", ")); at("Unit" + (n.units.length > 1 ? "s" : ""), n.units.join(", "));
at("Format", n.format); at("Format", n.format);
if(n.kind !== "DataElement") at("Physical name", n.physical);
} }
at("Activation", n.activation.replace(/([A-Z])/g," $1").trim()); at("Activation", n.activation.replace(/([A-Z])/g," $1").trim());
at("Version", n.version); at("Version", n.version);
@@ -1287,7 +1314,8 @@ function paint(id){
+ esc(r[1])+'</span></div>'; }).join("")); + esc(r[1])+'</span></div>'; }).join(""));
/* ---- 2. what it means ---- */ /* ---- 2. what it means ---- */
if(n.definition) h.push('<h4>Definition</h4><p>' + esc(n.definition) + '</p>'); h.push('<h4>What it is for</h4><p>' + (n.definition ? esc(n.definition)
: '<span class="mono">No definition yet.</span>') + '</p>');
if(n.formula) h.push('<h4>Calculation rule</h4><p class="mono">' if(n.formula) h.push('<h4>Calculation rule</h4><p class="mono">'
+ esc(n.formula) + '</p>'); + esc(n.formula) + '</p>');
if(n.rule) h.push('<h4>' + (n.kind === "DataContract" ? "Commitment" : "Open question") if(n.rule) h.push('<h4>' + (n.kind === "DataContract" ? "Commitment" : "Open question")
@@ -1304,7 +1332,10 @@ function paint(id){
else if(r[1] === "productOwner") ids = n.productOwner ? [n.productOwner] : []; else if(r[1] === "productOwner") ids = n.productOwner ? [n.productOwner] : [];
else if(Array.isArray(n[r[1]])) ids = n[r[1]]; else if(Array.isArray(n[r[1]])) ids = n[r[1]];
else if(n[r[1]]) ids = [n[r[1]]]; else if(n[r[1]]) ids = [n[r[1]]];
var names = ids.map(function(i){ return byId[i] ? esc(byId[i].name) : ""; }) /* actors carry a short label too; the ownership block shows it rather
than the full record name */
var names = ids.map(function(i){
return byId[i] ? esc(byId[i].shortName || byId[i].name) : ""; })
.filter(Boolean); .filter(Boolean);
var inherited = (r[1] === "steward_inherited" && n.stewardFrom && names.length); var inherited = (r[1] === "steward_inherited" && n.stewardFrom && names.length);
return '<div class="row"><span>' + r[0] + '</span><span class="mono">' return '<div class="row"><span>' + r[0] + '</span><span class="mono">'
@@ -1331,22 +1362,24 @@ function paint(id){
link("Business objects", BOS_OF[n.id]); link("Business objects", BOS_OF[n.id]);
link("Is about", [n.about]); link("Is about", [n.about]);
link("Also uses", n.uses); link("Also uses", n.uses);
/* a concept is the subject of the objects that point at it: shown before
the metrics, because the objects are the coarser grain */
link("Defines", ABOUT_OF[n.id]);
link("Used by", USES_OF[n.id]);
link("Metrics", (n.hasMetric || []).concat(METRICS_OF[n.id] || [])); link("Metrics", (n.hasMetric || []).concat(METRICS_OF[n.id] || []));
link("Measures", n.measures); link("Measures", n.measures);
link("Represents", [n.represents]); link("Represents", [n.represents]);
link("Implements metric", n.computedFrom); link("Implements", n.computedFrom);
link("Governed by", [n.governedBy]);
link("Exposes", n.exposes);
link("Packages", n.packages); link("Packages", n.packages);
link("Data objects", REALISED_BY[n.id]); link("Data objects", REALISED_BY[n.id]);
link("References", n.references); link("Qualified by", n.qualifiedBy);
link("Grain", n.grain); link("Grain", n.grain);
link("Held by data object", [PARENT_DO[n.id]]); link("Held by", [PARENT_DO[n.id]]);
link("Computed by", n.computedBy); link("Computed by", n.computedBy);
if(n.hasElement.length) links.push(["Data elements", n.hasElement]); if(n.hasElement.length) links.push(["Data elements", n.hasElement]);
link("Materialized as", [n.materializedAs]);
link("Stored in", [n.storedIn]); link("Stored in", [n.storedIn]);
link("Base table", [n.relation]); link("Sourced from", [n.sourcedFrom]);
link("Held by", [n.relation]);
link("Provides data for", BACKS[n.id]); link("Provides data for", BACKS[n.id]);
links.forEach(function(l){ links.forEach(function(l){
+142 -64
View File
@@ -13,7 +13,7 @@ Same visual language as the SODH viewer, with one substitution. Where SODH
lays its columns out by owning domain, the T-Box has no domains -- its lays its columns out by owning domain, the T-Box has no domains -- its
orthogonal axis is PROVENANCE: what governance asserts against what a orthogonal axis is PROVENANCE: what governance asserts against what a
harvester observes. The two-colour code carries that instead, blue for harvester observes. The two-colour code carries that instead, blue for
curated and dark gold for observed, and the seam is what crosses between them. defined and dark gold for captured, and the seam is what crosses between them.
No dependency: parses the Turtle subset this file uses, so it runs on the DSM No dependency: parses the Turtle subset this file uses, so it runs on the DSM
system python without a venv. system python without a venv.
@@ -33,8 +33,8 @@ WANTED = {
"rdfs:domain": "domain", "rdfs:range": "range", "rdfs:domain": "domain", "rdfs:range": "range",
"owl:inverseOf": "inverseOf", "owl:deprecated": "deprecated", "owl:inverseOf": "inverseOf", "owl:deprecated": "deprecated",
"dcterms:isReplacedBy": "isReplacedBy", "dcterms:isReplacedBy": "isReplacedBy",
"pr:harvestSource": "harvestSource", "pr:curationMode": "curationMode", "pr:harvestSource": "harvestSource", "pr:authoringMode": "authoringMode",
"pr:hasAcronym": "acronym", "pr:hasAcronym": "acronym", "pr:hasShortLabel": "shortLabel",
} }
LAYER_ROOT = OrderedDict([ LAYER_ROOT = OrderedDict([
@@ -44,7 +44,7 @@ LAYER_ROOT = OrderedDict([
("pr:LogicalLayerObject", ("logical", "Logical", ("pr:LogicalLayerObject", ("logical", "Logical",
"how it is structured")), "how it is structured")),
("pr:DeliveryLayerObject", ("delivery", "Delivery", ("pr:DeliveryLayerObject", ("delivery", "Delivery",
"what we promise, to whom")), "what we promise, to whom, until when")),
("pr:PhysicalLayerObject", ("physical", "Physical", ("pr:PhysicalLayerObject", ("physical", "Physical",
"what actually runs")), "what actually runs")),
("pr:ConsumptionLayerObject", ("consumption", "Consumption", ("pr:ConsumptionLayerObject", ("consumption", "Consumption",
@@ -213,19 +213,24 @@ def build(subjects):
layer = "vocab" layer = "vocab"
else: else:
layer = "roots" layer = "roots"
prov = ("observed" if "pr:ObservedObject" in anc # provenance is INFERRED from the type-of chain, never asserted:
else "curated" if "pr:CuratedObject" in anc else "structural") # a Sub-domain is Defined because it is a kind of Ownership Layer
# Object, which is a kind of Defined Object. No separate edge exists
# because none is needed -- the subclass chain already says it.
prov = ("captured" if "pr:CapturedObject" in anc
else "defined" if "pr:DefinedObject" in anc else "structural")
depth, p, guard = 0, parents.get(term), 0 depth, p, guard = 0, parents.get(term), 0
while p and p in classes and guard < 8: while p and p in classes and guard < 8:
depth += 1; guard += 1; p = parents.get(p) depth += 1; guard += 1; p = parents.get(p)
nodes[term] = { nodes[term] = {
"id": term, "kind": "Class", "id": term, "kind": "Class",
"label": first(rec, "label", term.split(":")[1]), "label": first(rec, "label", term.split(":")[1]),
"shortLabel": first(rec, "shortLabel"),
"layer": layer, "prov": prov, "depth": depth, "layer": layer, "prov": prov, "depth": depth,
"parent": parents.get(term, ""), "parent": parents.get(term, ""),
"comment": first(rec, "comment"), "comment": first(rec, "comment"),
"harvest": first(rec, "harvestSource"), "harvest": first(rec, "harvestSource"),
"curation": first(rec, "curationMode"), "authoring": first(rec, "authoringMode"),
"acronym": first(rec, "acronym"), "acronym": first(rec, "acronym"),
"deprecated": first(rec, "deprecated") in ("true", "True"), "deprecated": first(rec, "deprecated") in ("true", "True"),
"replacedBy": first(rec, "isReplacedBy"), "replacedBy": first(rec, "isReplacedBy"),
@@ -240,11 +245,12 @@ def build(subjects):
return { return {
"id": term, "kind": kind, "id": term, "kind": kind,
"label": first(rec, "label", term.split(":")[1]), "label": first(rec, "label", term.split(":")[1]),
"shortLabel": first(rec, "shortLabel"),
"comment": first(rec, "comment"), "comment": first(rec, "comment"),
"domain": first(rec, "domain"), "range": first(rec, "range"), "domain": first(rec, "domain"), "range": first(rec, "range"),
"inverseOf": first(rec, "inverseOf"), "inverseOf": first(rec, "inverseOf"),
"subPropertyOf": first(rec, "subPropertyOf"), "subPropertyOf": first(rec, "subPropertyOf"),
"curation": first(rec, "curationMode"), "authoring": first(rec, "authoringMode"),
"harvest": first(rec, "harvestSource"), "harvest": first(rec, "harvestSource"),
"functional": "owl:FunctionalProperty" in rec.get("type", []), "functional": "owl:FunctionalProperty" in rec.get("type", []),
"transitive": "owl:TransitiveProperty" in rec.get("type", []), "transitive": "owl:TransitiveProperty" in rec.get("type", []),
@@ -298,7 +304,7 @@ def main():
c = len([n for n in live_c if n["layer"] == l["id"]]) c = len([n for n in live_c if n["layer"] == l["id"]])
if c: if c:
print(" %-28s %d" % (l["title"], c)) print(" %-28s %d" % (l["title"], c))
for p in ("curated", "observed", "structural"): for p in ("defined", "captured", "structural"):
c = len([n for n in live_c if n["prov"] == p]) c = len([n for n in live_c if n["prov"] == p])
print(" %-28s %d" % (p, c)) print(" %-28s %d" % (p, c))
@@ -528,10 +534,33 @@ var G = {mode:"strata", provs:[], sel:null, hot:null, labels:true,
var NW=126, NH=32, LANEPAD=26, ROWGAP=15, BANDGAP=34, GUT=142; var NW=126, NH=32, LANEPAD=26, ROWGAP=15, BANDGAP=34, GUT=142;
function provColour(p){ function provColour(p){
return p === "observed" ? "#8A6A28" : p === "curated" ? "#1E4B9B" : "#6B6B6B"; return p === "captured" ? "#8A6A28" : p === "defined" ? "#1E4B9B" : "#6B6B6B";
} }
var LANES = [["curated","Curated \u00b7 asserted by governance","#1E4B9B"], /* Scoped relations: these carry no rdfs:domain or rdfs:range on purpose,
["observed","Observed \u00b7 harvested nightly","#8A6A28"], because each serves several classes with different targets and the scope
belongs in the SHACL shapes. Both the graph and the cards read this list,
so an edge drawn in one is never missing from the other. */
var SCOPED_PAIRS = [
["pr:belongsTo", "pr:SubDomain", "pr:DataDomain", "belongs to"],
["pr:belongsTo", "pr:BusinessObject", "pr:SubDomain", "belongs to"],
["pr:represents", "pr:DataObject", "pr:BusinessObject", "represents"],
["pr:represents", "pr:DataElement", "pr:BusinessConcept", "represents"],
["pr:monitoredBy", "pr:BusinessObject", "pr:DataSteward", "monitored by"],
["pr:owningDomain", "pr:DefinedObject", "pr:DataDomain", "owning domain"],
["pr:ownedBy", "pr:DefinedObject", "pr:Actor", "owned by"]
];
var ORDER = ["pr:MetaModelObject", "pr:DefinedObject", "pr:CapturedObject",
"pr:OwnershipLayerObject", "pr:DataDomain", "pr:SubDomain", "pr:BusinessObject",
"pr:BusinessLayerObject", "pr:BusinessConcept", "pr:Metric", "pr:KPI",
"pr:LogicalLayerObject", "pr:DataObject", "pr:DataElement",
"pr:DeliveryLayerObject", "pr:DataProduct", "pr:DataContract", "pr:DataInterface",
"pr:PhysicalLayerObject", "pr:System", "pr:Database", "pr:Schema",
"pr:DataStructure", "pr:BaseTable", "pr:View", "pr:ExternalTable", "pr:Field",
"pr:KeyConstraint", "pr:PrimaryKey", "pr:ForeignKeyConstraint", "pr:Transformation",
"pr:ConsumptionLayerObject", "pr:BIWorkspace", "pr:BIDataSource", "pr:BIField",
"pr:BIReport"];
var LANES = [["defined","Defined \u00b7 someone decided it","#1E4B9B"],
["captured","Captured \u00b7 read from a running system","#8A6A28"],
["structural","Structural","#6B6B6B"]]; ["structural","Structural","#6B6B6B"]];
function live(a){ return a.filter(function(x){ return !x.deprecated; }); } function live(a){ return a.filter(function(x){ return !x.deprecated; }); }
function layerTitle(id){ function layerTitle(id){
@@ -544,18 +573,43 @@ function buildGraph(){
live(M.nodes).forEach(function(n){ live(M.nodes).forEach(function(n){
if(!G.layers[n.layer]) return; if(!G.layers[n.layer]) return;
if(G.provs.length && G.provs.indexOf(n.prov) === -1) return; if(G.provs.length && G.provs.indexOf(n.prov) === -1) return;
/* row = rank inside the layer, so the hierarchy reads DOWNWARD:
layer root, then Domain, then Sub-domain, then Business object.
Depth alone put siblings side by side, which read left to right. */
var rank = ORDER.indexOf(n.id);
keep[n.id] = {n:n, x:0, y:0, keep[n.id] = {n:n, x:0, y:0,
layer: M.layers.map(function(l){return l.id;}).indexOf(n.layer), layer: M.layers.map(function(l){return l.id;}).indexOf(n.layer),
row: Math.min(n.depth, 2)}; row: rank > -1 ? rank : 90 + Math.min(n.depth, 5)};
}); });
var E = []; var E = [];
/* Properties deliberately left without rdfs:domain/range serve several
classes with different targets, and their scope lives in the SHACL
shapes instead. The graph reads domain and range, so those edges were
simply absent -- Business Object to Sub-domain among them. The pairs are
declared here, mirroring the shapes, rather than added to the T-Box where
they would over-constrain the property. */
SCOPED_PAIRS.forEach(function(sc){
if(byId[sc[0]] && !byId[sc[0]].deprecated && keep[sc[1]] && keep[sc[2]]
&& sc[1] !== sc[2])
E.push({s:sc[1], t:sc[2], l:sc[3], cross:false});
});
var _unused = [
["pr:belongsTo", "pr:SubDomain", "pr:DataDomain", "belongs to"],
["pr:belongsTo", "pr:BusinessObject", "pr:SubDomain", "belongs to"],
["pr:represents", "pr:DataObject", "pr:BusinessObject", "represents"],
["pr:represents", "pr:DataElement", "pr:BusinessConcept", "represents"],
["pr:monitoredBy", "pr:BusinessObject", "pr:DataSteward", "monitored by"],
["pr:owningDomain", "pr:DefinedObject", "pr:DataDomain", "owning domain"],
["pr:ownedBy", "pr:DefinedObject", "pr:Actor", "owned by"]
];
live(M.relations).forEach(function(r){ live(M.relations).forEach(function(r){
if(keep[r.domain] && keep[r.range] && r.domain !== r.range) if(keep[r.domain] && keep[r.range] && r.domain !== r.range)
E.push({s:r.domain, t:r.range, l:r.label, cross:r.seam}); E.push({s:r.domain, t:r.range, l:r.label, cross:r.seam});
}); });
live(M.nodes).forEach(function(n){ live(M.nodes).forEach(function(n){
if(n.parent && keep[n.id] && keep[n.parent]) if(n.parent && keep[n.id] && keep[n.parent])
E.push({s:n.id, t:n.parent, l:"specialises", cross:false}); E.push({s:n.id, t:n.parent, l:"type of", cross:false});
}); });
G.nodes = Object.keys(keep).map(function(k){ return keep[k]; }); G.nodes = Object.keys(keep).map(function(k){ return keep[k]; });
G.edges = E; G.edges = E;
@@ -569,19 +623,15 @@ function layoutGraph(idx){
if(G.mode === "network"){ layoutForce(idx); return; } if(G.mode === "network"){ layoutForce(idx); return; }
var lanes = LANES.map(function(L){ return L[0]; }) var lanes = LANES.map(function(L){ return L[0]; })
.filter(function(c){ return G.nodes.some(function(n){ return n.n.prov === c; }); }); .filter(function(c){ return G.nodes.some(function(n){ return n.n.prov === c; }); });
/* One class per row. A lane is one column wide, so a layer reads DOWNWARDS:
the layer class on top, then Domain, then Sub-domain, then Business
Object. That is how a hierarchy is read; laying them left to right made
the reader scan across for something that descends. */
var laneW = {}, x = GUT; var laneW = {}, x = GUT;
G._lanes = []; G._lanes = [];
lanes.forEach(function(code){ lanes.forEach(function(code){
var members = G.nodes.filter(function(n){ return n.n.prov === code; }); var w = NW + LANEPAD*2;
var widest = 1; laneW[code] = {x:x, w:w};
M.layers.forEach(function(_, li){
[0,1,2].forEach(function(ri){
var c = members.filter(function(n){ return n.layer===li && n.row===ri; }).length;
if(c) widest = Math.max(widest, Math.min(c, Math.ceil(Math.sqrt(c*1.6))));
});
});
var w = widest*(NW+12) + LANEPAD*2;
laneW[code] = {x:x, w:w, per:widest};
var L = LANES.filter(function(z){ return z[0] === code; })[0]; var L = LANES.filter(function(z){ return z[0] === code; })[0];
G._lanes.push({code:code, x:x, w:w, label:L[1], colour:L[2]}); G._lanes.push({code:code, x:x, w:w, label:L[1], colour:L[2]});
x += w; x += w;
@@ -591,30 +641,23 @@ function layoutGraph(idx){
var y = 0; var y = 0;
M.layers.forEach(function(b, li){ M.layers.forEach(function(b, li){
b._y = y; b._y = y;
var maxRows = 0; var any = G.nodes.filter(function(n){ return n.layer === li; });
var rowTops = {}; if(!any.length){ b._h = 0; return; }
var cursor = y + 34; var cursor = y + 36, rows = 0;
[0,1,2].forEach(function(ri){
var any = G.nodes.filter(function(n){ return n.layer===li && n.row===ri; });
if(!any.length){ rowTops[ri] = cursor; return; }
rowTops[ri] = cursor;
var deepest = 0;
lanes.forEach(function(code){ lanes.forEach(function(code){
var g = any.filter(function(n){ return n.n.prov === code; }); var g = any.filter(function(n){ return n.n.prov === code; });
if(!g.length) return; if(!g.length) return;
var per = laneW[code].per; g.sort(function(p, q){
deepest = Math.max(deepest, Math.ceil(g.length/per)); return ((ORDER.indexOf(p.n.id)+1)||99) - ((ORDER.indexOf(q.n.id)+1)||99)
g.sort(function(p,q){ return p.n.label.localeCompare(q.n.label); }); || p.n.label.localeCompare(q.n.label); });
g.forEach(function(n, i){ g.forEach(function(n, i){
n.x = laneW[code].x + LANEPAD + (i%per)*(NW+12) + NW/2; n.x = laneW[code].x + laneW[code].w/2;
n.y = cursor + Math.floor(i/per)*(NH+ROWGAP) + NH/2; n.y = cursor + i*(NH+ROWGAP) + NH/2;
}); });
rows = Math.max(rows, g.length);
}); });
cursor += deepest*(NH+ROWGAP) + ROWGAP; b._h = rows*(NH+ROWGAP) + 48;
maxRows += deepest; y += b._h + BANDGAP;
});
b._h = maxRows ? (cursor - y + 12) : 0;
y += b._h + (b._h ? BANDGAP : 0);
}); });
G._height = y; G._height = y;
} }
@@ -676,7 +719,7 @@ function renderGraph(){
if(G.mode === "strata" && G.nodes.length){ if(G.mode === "strata" && G.nodes.length){
/* domain lanes: alternating tint, so a column reads as one domain */ /* domain lanes: alternating tint, so a column reads as one domain */
(G._lanes||[]).forEach(function(l, i){ (G._lanes||[]).forEach(function(l, i){
var isSubject = l.code === "curated"; var isSubject = l.code === "defined";
bL.appendChild(el("rect", {x:l.x, y:-6, width:l.w, height:G._height+10, bL.appendChild(el("rect", {x:l.x, y:-6, width:l.w, height:G._height+10,
fill:l.colour, opacity:isSubject ? .05 : .028})); fill:l.colour, opacity:isSubject ? .05 : .028}));
if(i) bL.appendChild(el("line", {x1:l.x, y1:-6, x2:l.x, y2:G._height+4, if(i) bL.appendChild(el("line", {x1:l.x, y1:-6, x2:l.x, y2:G._height+4,
@@ -733,7 +776,7 @@ function renderGraph(){
specialisations a shape, never instantiated */ specialisations a shape, never instantiated */
g.appendChild(el("rect",{width:NW, height:NH, rx:3, fill:"#FFFFFF", g.appendChild(el("rect",{width:NW, height:NH, rx:3, fill:"#FFFFFF",
stroke:c, "stroke-dasharray": n.n.abstract ? "5 3" : "none"})); stroke:c, "stroke-dasharray": n.n.abstract ? "5 3" : "none"}));
var lines = wrapText(n.n.label, 21); var lines = wrapText(n.n.shortLabel || n.n.label, 21);
lines.forEach(function(t,i){ lines.forEach(function(t,i){
var e = el("text",{x:NW/2, y:NH/2+3.5+(i-(lines.length-1)/2)*10, fill:"#1A1A1A"}); var e = el("text",{x:NW/2, y:NH/2+3.5+(i-(lines.length-1)/2)*10, fill:"#1A1A1A"});
e.textContent = t; g.appendChild(e); e.textContent = t; g.appendChild(e);
@@ -807,33 +850,33 @@ function zoomG(f){
function kpi(v,l,w){ return '<div class="kpi'+(w?' warn':'')+'"><b>'+v+'</b><span>'+l+'</span></div>'; } function kpi(v,l,w){ return '<div class="kpi'+(w?' warn':'')+'"><b>'+v+'</b><span>'+l+'</span></div>'; }
function renderScope(){ function renderScope(){
var cls=live(M.nodes), rel=live(M.relations), att=live(M.attributes); var cls=live(M.nodes), rel=live(M.relations), att=live(M.attributes);
var cur=cls.filter(function(n){return n.prov==="curated";}); var cur=cls.filter(function(n){return n.prov==="defined";});
var obs=cls.filter(function(n){return n.prov==="observed";}); var obs=cls.filter(function(n){return n.prov==="captured";});
var str=cls.filter(function(n){return n.prov==="structural";}); var str=cls.filter(function(n){return n.prov==="structural";});
var seam=rel.filter(function(r){return r.seam;}); var seam=rel.filter(function(r){return r.seam;});
var dep=M.nodes.filter(function(n){return n.deprecated;}) var dep=M.nodes.filter(function(n){return n.deprecated;})
.concat(M.relations.concat(M.attributes).filter(function(r){return r.deprecated;})); .concat(M.relations.concat(M.attributes).filter(function(r){return r.deprecated;}));
var h=[]; var h=[];
h.push('<p class="lede">The layer says what an object <b>is</b>. The provenance says <b>how it got here</b>.</p>'); h.push('<p class="lede">The layer says what an object <b>is</b>. The provenance says <b>how it got here</b>.</p>');
h.push('<p class="sublede">Curated objects are asserted by governance and reviewed through a pull request. ' h.push('<p class="sublede">Defined objects are written by governance and reviewed through a pull request. '
+ 'Observed objects are harvested from Snowflake, Tableau and Looker, rebuilt on every run and never edited ' + 'Captured objects are read from Snowflake, Tableau and Looker, rebuilt on every run and never edited '
+ 'by hand. Without that split the ownership rules would demand an owner, a status and a version on every ' + 'by hand. Without that split the ownership rules would demand an owner, a status and a version on every '
+ 'one of some fifty thousand warehouse columns.</p>'); + 'one of some fifty thousand warehouse columns.</p>');
h.push('<div class="kpis">'+kpi(cls.length,"Classes")+kpi(rel.length,"Relations") h.push('<div class="kpis">'+kpi(cls.length,"Classes")+kpi(rel.length,"Relations")
+kpi(att.length,"Attributes")+kpi(M.bridges.length,"Bridge properties") +kpi(att.length,"Attributes")+kpi(M.bridges.length,"Bridge properties")
+kpi(seam.length,"Seam edges",true)+kpi(dep.length,"Deprecated",true)+'</div>'); +kpi(seam.length,"Seam edges",true)+kpi(dep.length,"Deprecated",true)+'</div>');
h.push('<h2 class="sec">Curated against observed</h2><div class="bar">' h.push('<h2 class="sec">Defined against captured</h2><div class="bar">'
+'<div style="width:'+(cur.length/cls.length*100)+'%;background:#1E4B9B"></div>' +'<div style="width:'+(cur.length/cls.length*100)+'%;background:#1E4B9B"></div>'
+'<div style="width:'+(obs.length/cls.length*100)+'%;background:#8A6A28"></div>' +'<div style="width:'+(obs.length/cls.length*100)+'%;background:#8A6A28"></div>'
+'<div style="width:'+(str.length/cls.length*100)+'%;background:#B9B4A7"></div></div>'); +'<div style="width:'+(str.length/cls.length*100)+'%;background:#B9B4A7"></div></div>');
h.push('<div class="legend">' h.push('<div class="legend">'
+'<span><i style="background:#1E4B9B"></i>Curated \u2014 '+cur.length+'</span>' +'<span><i style="background:#1E4B9B"></i>Defined \u2014 '+cur.length+'</span>'
+'<span><i style="background:#8A6A28"></i>Observed \u2014 '+obs.length+'</span>' +'<span><i style="background:#8A6A28"></i>Captured \u2014 '+obs.length+'</span>'
+'<span><i style="background:#B9B4A7"></i>Structural \u2014 '+str.length+'</span></div>'); +'<span><i style="background:#B9B4A7"></i>Structural \u2014 '+str.length+'</span></div>');
h.push('<div class="note" style="max-width:880px;margin-bottom:26px"><b>Only '+seam.length h.push('<div class="note" style="max-width:880px;margin-bottom:26px"><b>Only '+seam.length
+' relations cross between the two.</b> '+seam.map(function(r){return esc(r.label);}).join(" and ") +' relations cross between the two.</b> '+seam.map(function(r){return esc(r.label);}).join(" and ")
+' \u2014 both proposed by the reconciliation job and confirmed by a steward. Everywhere else the ' +' \u2014 both proposed by the reconciliation job and confirmed by a steward. Everywhere else the '
+'graph is either pure governance or pure machine, which is what keeps the curation cost bounded.</div>'); +'graph is either pure governance or pure machine, which is what keeps the mapping effort bounded.</div>');
h.push('<h2 class="sec">The layers</h2><div class="grid">'); h.push('<h2 class="sec">The layers</h2><div class="grid">');
M.layers.forEach(function(l){ M.layers.forEach(function(l){
var g=cls.filter(function(n){return n.layer===l.id;}); var g=cls.filter(function(n){return n.layer===l.id;});
@@ -861,9 +904,12 @@ function renderModel(){
M.layers.forEach(function(l){ M.layers.forEach(function(l){
var g=live(M.nodes).filter(function(n){return n.layer===l.id;}); var g=live(M.nodes).filter(function(n){return n.layer===l.id;});
if(!g.length) return; if(!g.length) return;
g.sort(function(a,b){ return a.depth-b.depth || a.label.localeCompare(b.label); }); g.sort(function(a,b){
return a.depth - b.depth
|| ((ORDER.indexOf(a.id)+1)||99) - ((ORDER.indexOf(b.id)+1)||99)
|| a.label.localeCompare(b.label); });
h.push('<h2 class="sec" style="color:'+provColour(g[0].prov)+'">'+esc(l.title)+' \u00b7 '+esc(l.tag)+'</h2>'); h.push('<h2 class="sec" style="color:'+provColour(g[0].prov)+'">'+esc(l.title)+' \u00b7 '+esc(l.tag)+'</h2>');
h.push('<table><thead><tr><th>Class</th><th>Provenance</th><th class="hide-s">Specialises</th>' h.push('<table><thead><tr><th>Class</th><th>Provenance</th><th class="hide-s">Type of</th>'
+'<th class="hide-s">What it is for</th></tr></thead><tbody>'); +'<th class="hide-s">What it is for</th></tr></thead><tbody>');
g.forEach(function(n){ g.forEach(function(n){
h.push('<tr data-id="'+n.id+'"><td><span class="dot" style="background:'+provColour(n.prov) h.push('<tr data-id="'+n.id+'"><td><span class="dot" style="background:'+provColour(n.prov)
@@ -903,7 +949,7 @@ function propRows(){
if(p.functional) t.push("exactly one"); if(p.functional) t.push("exactly one");
if(p.transitive) t.push("transitive"); if(p.transitive) t.push("transitive");
if(p.seam) t.push("seam"); if(p.seam) t.push("seam");
if(p.curation) t.push(p.curation.toLowerCase().replace(/_/g," ")); if(p.authoring) t.push(p.authoring.toLowerCase().replace(/_/g," "));
h.push('<tr data-id="'+p.id+'"><td><b>'+esc(p.label)+'</b><div class="mono">'+esc(p.id)+'</div></td>' h.push('<tr data-id="'+p.id+'"><td><b>'+esc(p.label)+'</b><div class="mono">'+esc(p.id)+'</div></td>'
+'<td class="hide-s">'+(byId[p.domain]?esc(clean(byId[p.domain].label)):'<span class="mono">any</span>')+'</td>' +'<td class="hide-s">'+(byId[p.domain]?esc(clean(byId[p.domain].label)):'<span class="mono">any</span>')+'</td>'
+'<td class="hide-s">'+(byId[p.range]?esc(clean(byId[p.range].label)) +'<td class="hide-s">'+(byId[p.range]?esc(clean(byId[p.range].label))
@@ -939,8 +985,8 @@ function renderDep(){
/* ------------------------------------------------------------ SHELL */ /* ------------------------------------------------------------ SHELL */
function renderGraphShell(){ function renderGraphShell(){
var h=['<p class="lede">The meta model as a graph.</p>' var h=['<p class="lede">The meta model as a graph.</p>'
+'<p class="sublede">Columns are provenance: <b style="color:#1E4B9B">curated</b> on the left, ' +'<p class="sublede">Columns are provenance: <b style="color:#1E4B9B">defined</b> on the left, '
+'<b style="color:#8A6A28">observed</b> on the right. Rows are layers. A <b>navy edge</b> crosses ' +'<b style="color:#8A6A28">captured</b> on the right. Rows are layers. A <b>navy edge</b> crosses '
+'between the two \u2014 that is the seam, the only place a steward confirms anything. Dashed outlines ' +'between the two \u2014 that is the seam, the only place a steward confirms anything. Dashed outlines '
+'are abstract classes, never instantiated.</p>']; +'are abstract classes, never instantiated.</p>'];
h.push('<div class="gwrap"><svg id="g-svg" xmlns="http://www.w3.org/2000/svg"><defs>' h.push('<div class="gwrap"><svg id="g-svg" xmlns="http://www.w3.org/2000/svg"><defs>'
@@ -1076,6 +1122,29 @@ M.relations.forEach(function(r){
if(r.deprecated) return; if(r.deprecated) return;
if(r.domain) (FROM[r.domain]=FROM[r.domain]||[]).push(r.id); if(r.domain) (FROM[r.domain]=FROM[r.domain]||[]).push(r.id);
if(r.range) (TO[r.range]=TO[r.range]||[]).push(r.id); }); if(r.range) (TO[r.range]=TO[r.range]||[]).push(r.id); });
/* The same scoped pairs the graph uses. Without them a Business Object shows
no incoming relation at all, because belongsTo and represents carry no
rdfs:domain -- their scope lives in the shapes. The card was silent about
edges the graph was drawing. */
SCOPED_PAIRS.forEach(function(sc){
if(!byId[sc[0]] || byId[sc[0]].deprecated) return;
(FROM[sc[1]] = FROM[sc[1]]||[]).push(sc[0]);
(TO[sc[2]] = TO[sc[2]]||[]).push(sc[0]);
});
/* the scoped properties again: without this a Sub-domain card showed no
incoming Business Object, and a Business Object card had no incoming list
at all, because neither relation declares a domain or a range */
[["pr:belongsTo","pr:SubDomain","pr:DataDomain"],
["pr:belongsTo","pr:BusinessObject","pr:SubDomain"],
["pr:represents","pr:DataObject","pr:BusinessObject"],
["pr:represents","pr:DataElement","pr:BusinessConcept"],
["pr:monitoredBy","pr:BusinessObject","pr:DataSteward"],
["pr:owningDomain","pr:DefinedObject","pr:DataDomain"],
["pr:ownedBy","pr:DefinedObject","pr:Actor"]].forEach(function(sc){
if(!byId[sc[0]] || byId[sc[0]].deprecated) return;
if((FROM[sc[1]]||[]).indexOf(sc[0])===-1) (FROM[sc[1]]=FROM[sc[1]]||[]).push(sc[0]);
if((TO[sc[2]]||[]).indexOf(sc[0])===-1) (TO[sc[2]]=TO[sc[2]]||[]).push(sc[0]);
});
M.attributes.forEach(function(a){ M.attributes.forEach(function(a){
if(a.deprecated||!a.domain) return; if(a.deprecated||!a.domain) return;
(ATTRS[a.domain]=ATTRS[a.domain]||[]).push(a.id); }); (ATTRS[a.domain]=ATTRS[a.domain]||[]).push(a.id); });
@@ -1104,7 +1173,10 @@ function paint(id){
: n.kind==="Attribute" ? "Attribute" : "Bridge property"; : n.kind==="Attribute" ? "Attribute" : "Bridge property";
h.push('<div class="kind" style="color:'+provColour(n.prov||"structural")+'">' h.push('<div class="kind" style="color:'+provColour(n.prov||"structural")+'">'
+ kindLabel + (n.layer ? ' \u00b7 ' + esc(layerTitle(n.layer)) : '') + '</div>'); + kindLabel + (n.layer ? ' \u00b7 ' + esc(layerTitle(n.layer)) : '') + '</div>');
h.push('<h3>'+esc(clean(n.label))+'</h3>'); h.push('<h3>'+esc(n.shortLabel || clean(n.label))+'</h3>');
if(n.shortLabel && n.shortLabel !== clean(n.label))
h.push('<div class="row" style="border:0;padding:0 0 6px"><span class="mono">'
+ esc(clean(n.label)) + '</span><span></span></div>');
h.push('<div class="iri">'+esc(n.id)+'</div>'); h.push('<div class="iri">'+esc(n.id)+'</div>');
if(n.deprecated) h.push('<div class="note"><b>Deprecated.</b> Kept so assertions written ' if(n.deprecated) h.push('<div class="note"><b>Deprecated.</b> Kept so assertions written '
+ 'against it keep classifying (LC-002)' + 'against it keep classifying (LC-002)'
@@ -1117,13 +1189,13 @@ function paint(id){
if(n.kind==="Class"){ if(n.kind==="Class"){
at("Provenance", n.prov); at("Provenance", n.prov);
at("Instantiated", n.abstract ? "never \u2014 abstract" : "yes"); at("Instantiated", n.abstract ? "never \u2014 abstract" : "yes");
at("Curation mode", (n.curation||"").toLowerCase().replace(/_/g," ")); at("Authoring mode", (n.authoring||"").toLowerCase().replace(/_/g," "));
at("Acronym", n.acronym); at("Acronym", n.acronym);
} else { } else {
at("Cardinality", n.functional ? "exactly one" : "many"); at("Cardinality", n.functional ? "exactly one" : "many");
at("Transitive", n.transitive ? "yes" : ""); at("Transitive", n.transitive ? "yes" : "");
at("Crosses the seam", n.seam ? "yes" : ""); at("Crosses the seam", n.seam ? "yes" : "");
at("Curation mode", (n.curation||"").toLowerCase().replace(/_/g," ")); at("Authoring mode", (n.authoring||"").toLowerCase().replace(/_/g," "));
if(!byId[n.range]) at("Range", n.range); if(!byId[n.range]) at("Range", n.range);
if(!n.domain) at("Domain", "any \u2014 scoped by SHACL per class"); if(!n.domain) at("Domain", "any \u2014 scoped by SHACL per class");
} }
@@ -1132,12 +1204,18 @@ function paint(id){
}).join("")); }).join(""));
/* 2. what it is for */ /* 2. what it is for */
if(n.comment) h.push('<h4>What it is for</h4><p>'+esc(n.comment).replace(/\n/g,"<br>")+'</p>'); h.push('<h4>What it is for</h4><p>' + (n.comment
? esc(n.comment).replace(/\n/g,"<br>")
: '<em style="color:var(--muted)">No definition yet \u2014 this term needs one '
+ 'before it can be published.</em>') + '</p>');
if(n.harvest) h.push('<h4>Harvested from</h4><div class="harv">'+esc(n.harvest)+'</div>'); if(n.harvest) h.push('<h4>Harvested from</h4><div class="harv">'+esc(n.harvest)+'</div>');
/* 3. related terms, from the coarsest down */ /* 3. related terms, from the coarsest down */
function link(label, ids){ function link(label, ids){
ids=(ids||[]).filter(function(i){return byId[i];}); /* deprecated terms live on their own tab; showing them in a link list
invites someone to use one */
ids=(ids||[]).filter(function(i){
return byId[i] && (!byId[i].deprecated || label === "Replaced by"); });
if(!ids.length) return; if(!ids.length) return;
h.push('<h4>'+label+'</h4>'); h.push('<h4>'+label+'</h4>');
ids.slice(0,16).forEach(function(i){ ids.slice(0,16).forEach(function(i){
@@ -1148,8 +1226,8 @@ function paint(id){
if(ids.length>16) h.push('<div class="row"><span class="mono">\u2026 and ' if(ids.length>16) h.push('<div class="row"><span class="mono">\u2026 and '
+(ids.length-16)+' more</span><span></span></div>'); +(ids.length-16)+' more</span><span></span></div>');
} }
link("Specialises", [n.parent, n.subPropertyOf]); link("Type of", [n.parent, n.subPropertyOf]);
link("Specialised by", CHILDREN[n.id]); link("Types", CHILDREN[n.id]);
link("Replaced by", [n.replacedBy]); link("Replaced by", [n.replacedBy]);
link("From", [n.domain]); link("From", [n.domain]);
link("To", [n.range]); link("To", [n.range]);
+148
View File
@@ -0,0 +1,148 @@
#!/usr/bin/env python3
"""
SODH - migration v1.2 -> v1.3
==============================
USAGE
python3 scripts/migrate_instances_v1_3.py # dry run
python3 scripts/migrate_instances_v1_3.py --apply # rewrite, .bak kept
1. references -> qualifiedBy. The name now says why the link exists rather
than how it is implemented.
2. hasSource removed from every Business Object. Same argument as for the
Business Concepts in v1.2, and the values showed why it was needed: across
the 15 objects the property held physical locations ("SODH Gold /
F_SELL_OUT"), migration notes ("MDM Product -> SODH [concepts TBC by
DD-10]") and arbitration remarks ("DGO proposal, pending ratification"),
all under one name. A Business Object is a governance grouping; its data
lives in the Data Objects that represent it, and those carry the system.
3. Short labels where the long name is unwieldy on screen. The Data Product
was called "Sell Out Data Hub - Gold" on its own card and "SODH Gold"
everywhere else, because the viewers each kept a display dictionary.
4. Instance versions aligned on the model version. Business Objects were
still stamped 0.6 while the model had moved to 1.3.
"""
import os
import re
import shutil
import sys
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
TTL = os.path.join(REPO, "instances", "sodh.ttl")
W = 78
SHORT_LABELS = {
"ex:DP_06_01_001": "SODH Gold",
"ex:DC_06_01_001": "SODH Gold contract",
"ex:DI_06_01_001": "Consumption schema",
"ex:DI_06_01_002": "Panel extract",
"ex:DO_06_01_001": "Sell Out Volume",
"ex:DO_06_01_003": "Sell Out Distribution",
"ex:DO_10_01_001": "Product",
"ex:DO_04_01_001": "Customer",
"ex:DO_16_01_001": "Currency",
"ex:DO_21_01_001": "Calendar",
}
MODEL_VERSION = "1.3"
def clauses(body):
out, buf, i, n = [], [], 0, len(body)
while i < n:
c = body[i]
if c == '"':
j = i + 1
while j < n and body[j] != '"':
j += 2 if body[j] == "\\" else 1
j = min(j + 1, n)
buf.append(body[i:j]); i = j; continue
if c == ";":
out.append("".join(buf)); buf = []; i += 1; continue
buf.append(c); i += 1
out.append("".join(buf))
return out
def set_prop(text, subject, prop, value):
pat = re.compile(r'(^%s a pr:\w+ ;)(.*?)(\.\s*\n)' % re.escape(subject), re.S | re.M)
m = pat.search(text)
if not m:
return text, False
kept = [c for c in clauses(m.group(2))
if c.strip() and not re.match(r'\s*%s\s' % re.escape(prop), c)]
if value is not None:
kept.insert(0, "\n %s %s " % (prop, value))
body = " ;".join(kept).rstrip() + "\n "
return text[:m.start()] + m.group(1) + body + m.group(3) + text[m.end():], True
def has_prop(text, subject, prop):
m = re.search(r'^%s a pr:\w+ ;(.*?)\.\s*\n' % re.escape(subject), text, re.S | re.M)
return bool(m) and any(re.match(r'\s*%s\s' % re.escape(prop), c)
for c in clauses(m.group(1)))
def main():
apply_changes = "--apply" in sys.argv
if not os.path.exists(TTL):
print("Not found: %s" % TTL); sys.exit(2)
text = original = open(TTL, encoding="utf-8").read()
report = []
text, n = re.subn(r'\bpr:references\b', 'pr:qualifiedBy', text)
report.append("references -> qualifiedBy: %d occurrence(s)" % n)
n_bo = 0
for m in list(re.finditer(r'^(ex:BO_\w+) a pr:BusinessObject ;', text, re.M)):
if has_prop(text, m.group(1), "pr:hasSource"):
text, _ = set_prop(text, m.group(1), "pr:hasSource", None)
n_bo += 1
report.append("hasSource removed from %d business objects" % n_bo)
n_sl = 0
for subj, label in SHORT_LABELS.items():
if re.search(r'^%s a ' % re.escape(subj), text, re.M):
text, ok = set_prop(text, subj, "pr:hasShortLabel", '"%s"' % label)
n_sl += ok
report.append("short label set on %d objects" % n_sl)
n_v = 0
for m in list(re.finditer(r'^(ex:\w+) a pr:\w+ ;', text, re.M)):
if has_prop(text, m.group(1), "pr:hasVersion"):
text, ok = set_prop(text, m.group(1), "pr:hasVersion",
'"%s"' % MODEL_VERSION)
n_v += ok
report.append("version aligned on %s for %d objects" % (MODEL_VERSION, n_v))
print()
print("SODH MIGRATION v1.2 -> v1.3 %s" % ("APPLY" if apply_changes else "DRY RUN"))
print("=" * W)
for line in report:
print(" " + line)
print("=" * W)
left_bo = len([1 for m in re.finditer(r'^(ex:BO_\w+) a pr:BusinessObject ;', text, re.M)
if has_prop(text, m.group(1), "pr:hasSource")])
doubled = []
for m in re.finditer(r'^(ex:\w+) a pr:\w+ ;(.*?)\.\s*\n', text, re.S | re.M):
for p in ("pr:hasName", "pr:hasShortLabel", "pr:hasVersion", "pr:hasSource"):
if len(re.findall(r'(?<![\w:])%s\s' % re.escape(p), m.group(2))) > 1:
doubled.append("%s/%s" % (m.group(1), p))
print(" business objects still carrying a source: %d (must be 0)" % left_bo)
print(" references left: %d (must be 0)" % len(re.findall(r'pr:references', text)))
print(" duplicated single-valued properties: %s"
% (", ".join(doubled) if doubled else "none"))
print("=" * W)
if text != original and apply_changes:
shutil.copy2(TTL, TTL + ".bak")
open(TTL, "w", encoding="utf-8").write(text)
print(" written, backup at %s.bak" % os.path.basename(TTL))
elif text != original:
print(" dry run -- re-run with --apply to write")
print()
if __name__ == "__main__":
main()
+176
View File
@@ -0,0 +1,176 @@
#!/usr/bin/env python3
"""
SODH - migration v1.3 -> v1.4
==============================
USAGE
python3 scripts/migrate_instances_v1_4.py # dry run
python3 scripts/migrate_instances_v1_4.py --apply # rewrite, .bak kept
1. SYSTEMS BECOME NODES. hasSource held a string: "SODH Gold", "MDM Product",
"Group Finance". Six systems are created and sourcedFrom points at them.
"Which objects depend on MDM Product?" was unanswerable while MDM Product
was text.
The systems are seeded here by hand, once. Everything below them --
databases, schemas, tables, columns -- is meant to be harvested, and this
script does not invent any of it.
2. Three hasSource values were not systems at all but migration notes ("DGO
proposal, split of a compound sub-domain"). They are dropped rather than
turned into a system: provenance of a record is not a property of the thing
the record describes.
3. materializedAs -> storedIn on the Data Objects. Nothing carries it yet,
done for completeness so the term never reappears.
"""
import os
import re
import shutil
import sys
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
TTL = os.path.join(REPO, "instances", "sodh.ttl")
W = 78
# (iri, identifier, name, short label, system type, the string it replaces)
SYSTEMS = [
("ex:SYS_SODH_GOLD", "SYS-001", "Sell Out Data Hub - Gold layer", "SODH Gold",
"pr:DataPlatform", "SODH Gold"),
("ex:SYS_MDM_PRODUCT", "SYS-002", "Master Data Management - Product", "MDM Product",
"pr:MasterDataSystem", "MDM Product"),
("ex:SYS_MDM_CUSTOMER", "SYS-003", "Master Data Management - Customer", "MDM Customer",
"pr:MasterDataSystem", "MDM Customer"),
("ex:SYS_GROUP_FINANCE", "SYS-004", "Group Finance consolidation", "Group Finance",
"pr:SourceApplication", "Group Finance"),
("ex:SYS_CORPORATE_CALENDAR", "SYS-005", "Corporate standard calendar", "Standard Calendar",
"pr:SourceApplication", "Standard Calendar"),
]
# strings that were never a system: provenance notes about the record itself
NOT_A_SYSTEM = ["DGO proposal"]
def clauses(body):
out, buf, i, n = [], [], 0, len(body)
while i < n:
c = body[i]
if c == '"':
j = i + 1
while j < n and body[j] != '"':
j += 2 if body[j] == "\\" else 1
j = min(j + 1, n)
buf.append(body[i:j]); i = j; continue
if c == ";":
out.append("".join(buf)); buf = []; i += 1; continue
buf.append(c); i += 1
out.append("".join(buf))
return out
def set_prop(text, subject, prop, value):
pat = re.compile(r'(^%s a pr:\w+ ;)(.*?)(\.\s*\n)' % re.escape(subject), re.S | re.M)
m = pat.search(text)
if not m:
return text, False
kept = [c for c in clauses(m.group(2))
if c.strip() and not re.match(r'\s*%s\s' % re.escape(prop), c)]
if value is not None:
kept.insert(0, "\n %s %s " % (prop, value))
body = " ;".join(kept).rstrip() + "\n "
return text[:m.start()] + m.group(1) + body + m.group(3) + text[m.end():], True
def get_source(text, subject):
m = re.search(r'^%s a pr:\w+ ;(.*?)\.\s*\n' % re.escape(subject), text, re.S | re.M)
if not m:
return None
for c in clauses(m.group(1)):
mm = re.match(r'\s*pr:hasSource\s+"([^"]*)"', c)
if mm:
return mm.group(1)
return None
def system_blocks(todo):
head = ("# --- systems -----------------------------------------------------------\n"
"# Seeded by hand, once. A system was a string until now, which is why\n"
"# 'what depends on MDM Product' had no answer. Everything below a system --\n"
"# databases, schemas, tables, columns -- is meant to be harvested, and\n"
"# nothing here invents any of it.\n\n")
out = []
for iri, ident, name, short, stype, _ in todo:
out.append('%s a pr:System ;\n'
' pr:hasIdentifier "%s" ; pr:hasName "%s" ;\n'
' pr:hasShortLabel "%s" ;\n'
' pr:hasSystemType %s ;\n'
' pr:hasStatus "DRAFT" ; pr:hasVersion "1.4" .'
% (iri, ident, name, short, stype))
return head + "\n".join(out) + "\n"
def main():
apply_changes = "--apply" in sys.argv
if not os.path.exists(TTL):
print("Not found: %s" % TTL); sys.exit(2)
text = original = open(TTL, encoding="utf-8").read()
report = []
todo = [sy for sy in SYSTEMS if not re.search(r'^%s a ' % re.escape(sy[0]), text, re.M)]
if todo:
anchor = re.search(r'^ex:DD_\w+ a pr:DataDomain', text, re.M)
pos = anchor.start() if anchor else 0
text = text[:pos] + system_blocks(todo) + "\n" + text[pos:]
report.append("%d systems created%s" % (len(todo), "" if todo else " (already present)"))
by_string = {sy[5]: sy[0] for sy in SYSTEMS}
n_ok = n_drop = 0
for m in list(re.finditer(r'^(ex:\w+) a pr:\w+ ;', text, re.M)):
subj = m.group(1)
src = get_source(text, subj)
if src is None:
continue
if src in by_string:
text, _ = set_prop(text, subj, "pr:hasSource", None)
text, _ = set_prop(text, subj, "pr:sourcedFrom", by_string[src])
n_ok += 1
elif any(k in src for k in NOT_A_SYSTEM):
text, _ = set_prop(text, subj, "pr:hasSource", None)
n_drop += 1
report.append("sourcedFrom set on %d objects" % n_ok)
report.append("%d provenance notes dropped rather than turned into a system" % n_drop)
text, n = re.subn(r'\bpr:materializedAs\b', 'pr:storedIn', text)
report.append("materializedAs -> storedIn: %d occurrence(s)" % n)
print()
print("SODH MIGRATION v1.3 -> v1.4 %s" % ("APPLY" if apply_changes else "DRY RUN"))
print("=" * W)
for line in report:
print(" " + line)
print("=" * W)
left = re.findall(r'pr:hasSource\s+"([^"]*)"', text)
defined = set(re.findall(r'^(ex:\w+) a ', text, re.M))
dangling = sorted(set(re.findall(r'\bex:\w+', text)) - defined)
doubled = []
for m in re.finditer(r'^(ex:\w+) a pr:\w+ ;(.*?)\.\s*\n', text, re.S | re.M):
for p in ("pr:sourcedFrom", "pr:hasName", "pr:hasShortLabel"):
if len(re.findall(r'(?<![\w:])%s\s' % re.escape(p), m.group(2))) > 1:
doubled.append("%s/%s" % (m.group(1), p))
print(" hasSource left: %d %s" % (len(left), sorted(set(left)) if left else "(must be 0)"))
print(" sourcedFrom references: %d" % len(re.findall(r'pr:sourcedFrom', text)))
print(" dangling references: %s" % (", ".join(dangling) if dangling else "none"))
print(" duplicated single-valued properties: %s"
% (", ".join(doubled) if doubled else "none"))
print("=" * W)
if text != original and apply_changes:
shutil.copy2(TTL, TTL + ".bak")
open(TTL, "w", encoding="utf-8").write(text)
print(" written, backup at %s.bak" % os.path.basename(TTL))
elif text != original:
print(" dry run -- re-run with --apply to write")
print()
if __name__ == "__main__":
main()
+59 -30
View File
@@ -13,8 +13,8 @@
# ---------------------- # ----------------------
# 1. SPLIT BY PROVENANCE. v0.1.1 targeted pr:MetaModelObject and demanded an # 1. SPLIT BY PROVENANCE. v0.1.1 targeted pr:MetaModelObject and demanded an
# identifier, a name, an owner, an owning domain, a status and a version on # identifier, a name, an owner, an owning domain, a status and a version on
# everything. Applied to the harvested half of the graph that would fail on # everything. Applied to the captured half of the graph that would fail on
# every one of ~50 000 Snowflake columns. Curated and Observed objects now # every one of ~50 000 Snowflake columns. Defined and Captured objects now
# have separate shapes with separate expectations. # have separate shapes with separate expectations.
# #
# 2. SCOPE FOR THE UNIFIED RELATIONS. belongsTo and represents deliberately # 2. SCOPE FOR THE UNIFIED RELATIONS. belongsTo and represents deliberately
@@ -40,29 +40,29 @@
<https://ontology.pernod-ricard.com/metamodel/shapes/> <https://ontology.pernod-ricard.com/metamodel/shapes/>
dcterms:title "PR Data Meta Model - SHACL Shapes" ; dcterms:title "PR Data Meta Model - SHACL Shapes" ;
dcterms:modified "2026-07-27"^^xsd:date ; dcterms:modified "2026-07-30"^^xsd:date ;
rdfs:comment "Executable form of the rulebook. Validated by scripts/run_shacl_validation.py." . rdfs:comment "Executable form of the rulebook. Validated by scripts/run_shacl_validation.py." .
################################################################# #################################################################
# IDENTITY AND LIFECYCLE -- curated objects only # IDENTITY AND LIFECYCLE -- defined objects only
################################################################# #################################################################
prs:CuratedObjectShape a sh:NodeShape ; prs:DefinedObjectShape a sh:NodeShape ;
sh:targetClass pr:CuratedObject ; sh:targetClass pr:DefinedObject ;
rdfs:comment "Applies to everything governance asserts. Observed objects are deliberately exempt: see ObservedObjectShape." ; rdfs:comment "Applies to everything governance writes. Captured objects are deliberately exempt: see CapturedObjectShape." ;
sh:property [ sh:property [
sh:path pr:hasIdentifier ; sh:datatype xsd:string ; sh:path pr:hasIdentifier ; sh:datatype xsd:string ;
sh:minCount 1 ; sh:maxCount 1 ; sh:minCount 1 ; sh:maxCount 1 ;
sh:severity sh:Violation ; sh:severity sh:Violation ;
sh:message "DQ-001 -- every curated object carries exactly one identifier. NOTE: uniqueness across the graph is NOT tested here; SHACL checks cardinality only. The reconciliation job tests uniqueness." ] ; sh:message "DQ-001 -- every defined object carries exactly one identifier. NOTE: uniqueness across the graph is NOT tested here; SHACL checks cardinality only. The reconciliation job tests uniqueness." ] ;
sh:property [ sh:property [
sh:path pr:hasName ; sh:datatype xsd:string ; sh:path pr:hasName ; sh:datatype xsd:string ;
sh:minCount 1 ; sh:maxCount 1 ; sh:minLength 3 ; sh:minCount 1 ; sh:maxCount 1 ; sh:minLength 3 ;
sh:severity sh:Violation ; sh:severity sh:Violation ;
sh:message "Every curated object carries exactly one canonical name." ] ; sh:message "Every defined object carries exactly one canonical name." ] ;
sh:property [ sh:property [
sh:path pr:hasStatus ; sh:minCount 1 ; sh:path pr:hasStatus ; sh:minCount 1 ;
@@ -82,7 +82,7 @@ prs:CuratedObjectShape a sh:NodeShape ;
prs:PublishedShape a sh:NodeShape ; prs:PublishedShape a sh:NodeShape ;
sh:targetClass pr:CuratedObject ; sh:targetClass pr:DefinedObject ;
rdfs:comment "Extra demands that only bite once an object claims to be PUBLISHED. Keeping these off the base shape is what lets a domain be documented progressively." ; rdfs:comment "Extra demands that only bite once an object claims to be PUBLISHED. Keeping these off the base shape is what lets a domain be documented progressively." ;
sh:sparql [ sh:sparql [
@@ -107,11 +107,11 @@ prs:PublishedShape a sh:NodeShape ;
################################################################# #################################################################
# OBSERVED OBJECTS -- harvested, never hand-edited # OBSERVED OBJECTS -- captured, never hand-edited
################################################################# #################################################################
prs:ObservedObjectShape a sh:NodeShape ; prs:CapturedObjectShape a sh:NodeShape ;
sh:targetClass pr:ObservedObject ; sh:targetClass pr:CapturedObject ;
rdfs:comment "No identifier, no name, no owner, no status, no version. A harvested column is attributed through the Data Element mapped to it, never asserted directly. What it must carry is enough to be re-found on the next run." ; rdfs:comment "No identifier, no name, no owner, no status, no version. A harvested column is attributed through the Data Element mapped to it, never asserted directly. What it must carry is enough to be re-found on the next run." ;
sh:property [ sh:property [
@@ -124,12 +124,26 @@ prs:ObservedObjectShape a sh:NodeShape ;
sh:path pr:harvestedOn ; sh:datatype xsd:dateTime ; sh:path pr:harvestedOn ; sh:datatype xsd:dateTime ;
sh:minCount 1 ; sh:maxCount 1 ; sh:minCount 1 ; sh:maxCount 1 ;
sh:severity sh:Violation ; sh:severity sh:Violation ;
sh:message "Every observed object records when it was harvested. Without it there is no way to tell a current fact from a stale one." ] ; sh:message "Every captured object records when it was harvested. Without it there is no way to tell a current fact from a stale one." ] ;
sh:property [ sh:property [
sh:path pr:owningDomain ; sh:maxCount 0 ; sh:path pr:owningDomain ; sh:maxCount 0 ;
sh:severity sh:Warning ; sh:severity sh:Warning ;
sh:message "Observed objects are not attributed directly -- ownership flows from the Data Element mapped to them." ] . sh:message "Captured objects are not attributed directly -- ownership flows from the Data Element mapped to them." ] .
# inSchema is deliberately NOT required anywhere. Neo4j has no schema and
# BigQuery calls the level a dataset; a minCount 1 here would break on the
# first harvest outside Snowflake.
prs:DataStructureShape a sh:NodeShape ;
sh:targetClass pr:DataStructure ;
sh:property [
sh:path pr:inSchema ; sh:maxCount 1 ;
sh:severity sh:Info ;
sh:message "Optional on purpose: not every engine has a schema level." ] ;
sh:property [
sh:path pr:sourcedFrom ; sh:maxCount 1 ;
sh:severity sh:Info ] .
prs:FieldShape a sh:NodeShape ; prs:FieldShape a sh:NodeShape ;
@@ -211,7 +225,12 @@ prs:BusinessObjectShape a sh:NodeShape ;
sh:property [ sh:property [
sh:path pr:hasBusinessDefinition ; sh:maxCount 0 ; sh:path pr:hasBusinessDefinition ; sh:maxCount 0 ;
sh:severity sh:Violation ; sh:severity sh:Violation ;
sh:message "A Business Object carries no definition of its own: meaning belongs to the Business Concept named by aboutConcept. Two definitions for one notion are two definitions free to diverge. MIGRATION: move the text to that Concept." ] . sh:message "A Business Object carries no definition of its own: meaning belongs to the Business Concept named by aboutConcept. Two definitions for one notion are two definitions free to diverge. MIGRATION: move the text to that Concept." ] ;
sh:property [
sh:path pr:sourcedFrom ; sh:maxCount 0 ;
sh:severity sh:Violation ;
sh:message "A Business Object has no system of truth. It is a governance grouping, not data: its data lives in the Data Objects that represent it, and those name the system. In v0.6 this property held a grab-bag on the 15 objects -- physical locations, migration notes, arbitration remarks -- all under one name." ] .
################################################################# #################################################################
@@ -234,9 +253,9 @@ prs:BusinessConceptShape a sh:NodeShape ;
sh:message "BR-003 -- the presence of a formula is what makes something a Metric. A Concept with a formula is a misclassified Metric." ] ; sh:message "BR-003 -- the presence of a formula is what makes something a Metric. A Concept with a formula is a misclassified Metric." ] ;
sh:property [ sh:property [
sh:path pr:hasSource ; sh:maxCount 0 ; sh:path pr:sourcedFrom ; sh:maxCount 0 ;
sh:severity sh:Violation ; sh:severity sh:Violation ;
sh:message "A Business Concept has no source of truth. Its authority is its owning domain and its definition belongs in the glossary. Naming a system here would ask which of the systems that USE the concept defines it -- a question with no answer." ] . sh:message "A Business Concept has no system of truth. Its authority is its owning domain and its definition belongs in the glossary. Naming a system here would ask which of the systems that USE the concept defines it -- a question with no answer." ] .
prs:MetricShape a sh:NodeShape ; prs:MetricShape a sh:NodeShape ;
@@ -277,27 +296,28 @@ prs:MetricShape a sh:NodeShape ;
# will drift the day an element moves. # will drift the day an element moves.
prs:MetricSourceShape a sh:NodeShape ; prs:MetricSourceShape a sh:NodeShape ;
sh:targetClass pr:Metric ; sh:targetClass pr:Metric ;
rdfs:comment "Rewritten in v1.4 to compare System references rather than strings. A string comparison would have called 'SODH Gold' and 'Snowflake (SODH Gold)' two different systems." ;
sh:sparql [ sh:sparql [
sh:severity sh:Violation ; sh:severity sh:Violation ;
sh:message "A Metric's source of truth must match the source of at least one element that computes it. If they disagree, either the metric points at the wrong system or an element has moved." ; sh:message "A Metric's system of truth must match that of at least one element computing it. If they disagree, either the metric points at the wrong system or an element has moved." ;
sh:select """ sh:select """
PREFIX pr: <https://ontology.pernod-ricard.com/metamodel/> PREFIX pr: <https://ontology.pernod-ricard.com/metamodel/>
SELECT $this WHERE { SELECT $this WHERE {
$this pr:hasSource ?src ; pr:computedBy ?de . $this pr:sourcedFrom ?sys ; pr:computedBy ?de .
FILTER NOT EXISTS { FILTER NOT EXISTS {
$this pr:computedBy ?any . $this pr:computedBy ?any .
?any pr:hasSource ?src . ?any pr:sourcedFrom ?sys .
} }
}""" ] ; }""" ] ;
sh:sparql [ sh:sparql [
sh:severity sh:Warning ; sh:severity sh:Warning ;
sh:message "Elements computing one Metric come from several systems of truth. Legitimate for a metric blending sources, worth a look otherwise." ; sh:message "Elements computing one Metric come from several systems. Legitimate for a metric blending sources, worth a look otherwise." ;
sh:select """ sh:select """
PREFIX pr: <https://ontology.pernod-ricard.com/metamodel/> PREFIX pr: <https://ontology.pernod-ricard.com/metamodel/>
SELECT $this WHERE { SELECT $this WHERE {
$this pr:computedBy ?a , ?b . $this pr:computedBy ?a , ?b .
?a pr:hasSource ?sa . ?a pr:sourcedFrom ?sa .
?b pr:hasSource ?sb . ?b pr:sourcedFrom ?sb .
FILTER(?sa != ?sb) FILTER(?sa != ?sb)
}""" ] . }""" ] .
@@ -346,10 +366,19 @@ prs:DataObjectShape a sh:NodeShape ;
sh:message "Stewardship is declared on the Business Object and inherited through represents. Declaring it here allows two answers to the same question." ] ; sh:message "Stewardship is declared on the Business Object and inherited through represents. Declaring it here allows two answers to the same question." ] ;
sh:property [ sh:property [
sh:path pr:hasSource ; sh:maxCount 1 ; sh:path pr:sourcedFrom ; sh:class pr:System ; sh:maxCount 1 ;
sh:not [ sh:pattern "/" ] ;
sh:severity sh:Warning ; sh:severity sh:Warning ;
sh:message "hasSource names the SYSTEM of origin, nothing else. A value like 'SODH Gold / F_SELL_OUT' packs a system and a physical relation into one string -- the physical relation belongs in physicalName." ] . sh:message "Name the System holding the authoritative data, as a reference rather than a string. A string could not answer 'what depends on MDM Product'." ] ;
sh:property [
sh:path pr:storedIn ; sh:class pr:DataStructure ;
sh:severity sh:Warning ;
sh:message "Where the object actually sits: a table, a view, later a node label. The seam between what governance defines and what a harvester captures -- proposed by the reconciliation job, confirmed by a steward." ] ;
sh:property [
sh:path pr:materializedAs ; sh:maxCount 0 ;
sh:severity sh:Warning ;
sh:message "Deprecated in v1.4: storedIn now serves both granularities." ] .
# A declared grain element must actually belong to the object it grains. # A declared grain element must actually belong to the object it grains.
@@ -411,7 +440,7 @@ prs:DataElementMeaningShape a sh:NodeShape ;
sh:minCount 1 ] ] sh:minCount 1 ] ]
) ; ) ;
sh:severity sh:Violation ; sh:severity sh:Violation ;
sh:message "A Data Element must reach business meaning by exactly one route: represents a Business Concept (dimensional), or be computedBy a Metric (measure). Never both, never neither. A curated column with no business meaning is debt, not an acceptable state." . sh:message "A Data Element must reach business meaning by exactly one route: represents a Business Concept (dimensional), or be computedBy a Metric (measure). Never both, never neither. A defined column with no business meaning is debt, not an acceptable state." .
################################################################# #################################################################
@@ -466,7 +495,7 @@ prs:BIFieldShape a sh:NodeShape ;
sh:targetClass pr:BIField ; sh:targetClass pr:BIField ;
rdfs:comment "realizesMetric is a transitional control: asserted directly, with no rule written against the lineage chain. Making the assertion conditional on intermediate mappings existing would create confusion and discourage people from filling it in at all." ; rdfs:comment "realizesMetric is a transitional control: asserted directly, with no rule written against the lineage chain. Making the assertion conditional on intermediate mappings existing would create confusion and discourage people from filling it in at all." ;
sh:property [ sh:property [
sh:path pr:realizesMetric ; sh:class pr:Metric ; sh:path pr:exposesMetric ; sh:class pr:Metric ;
sh:maxCount 1 ; sh:maxCount 1 ;
sh:severity sh:Info ; sh:severity sh:Info ;
sh:message "Which certified Metric this BI field implements. Comparing expressions across fields realizing one Metric is the harmonisation measure." ] . sh:message "Which certified Metric this BI field implements. Comparing expressions across fields realizing one Metric is the harmonisation measure." ] .
@@ -477,7 +506,7 @@ prs:BIFieldShape a sh:NodeShape ;
################################################################# #################################################################
prs:NamingShape a sh:NodeShape ; prs:NamingShape a sh:NodeShape ;
sh:targetClass pr:CuratedObject ; sh:targetClass pr:DefinedObject ;
sh:property [ sh:property [
sh:path pr:hasName ; sh:path pr:hasName ;
sh:not [ sh:pattern "_NM$" ] ; sh:not [ sh:pattern "_NM$" ] ;