Compare commits
4 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Bastien Gourdon
|
76e95de471 | ||
|
Bastien Gourdon
|
7793def268 | ||
|
Bastien Gourdon
|
45ca9ffb68 | ||
|
Bastien Gourdon
|
c8693b63bb |
+831
-550
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+1382
-364
File diff suppressed because it is too large
Load Diff
+301
-70
@@ -7,11 +7,113 @@
|
||||
@prefix pr: <https://ontology.pernod-ricard.com/metamodel/> .
|
||||
|
||||
#################################################################
|
||||
# PERNOD RICARD DATA META MODEL - T-Box v1.0
|
||||
# PERNOD RICARD DATA META MODEL - T-Box v1.2
|
||||
# Owner: Bastien Gourdon - Data Governance Office
|
||||
#
|
||||
# WHAT CHANGED vs v0.9.2
|
||||
# ----------------------
|
||||
# WHAT CHANGED vs v1.1
|
||||
# --------------------
|
||||
# A. physicalizedIn -> storedIn. "Physicalized in" is jargon; the relation
|
||||
# says where the data sits, and "stored in" says that to anyone.
|
||||
#
|
||||
# B. hasSource MEANS ONE THING: the system that holds the authoritative data.
|
||||
# It was drifting into a third meaning on Business Concepts and Metrics,
|
||||
# where it named the DOCUMENT a record came from ("SODH back-doc v0.7").
|
||||
# A business concept has no data source -- it has an owning domain, which
|
||||
# is its authority, and a definition published in the glossary. The
|
||||
# property is now forbidden on concepts by SHACL.
|
||||
#
|
||||
# C. FIVE TERMS REMOVED OR DEPRECATED, none of them instantiated anywhere:
|
||||
# Table deleted -- deprecated since v1.0, replaced by
|
||||
# BaseTable, and never used. A deprecation that
|
||||
# never ends stops meaning anything.
|
||||
# constrainsRelation derivable: constrainsField -> livesIn gives the
|
||||
# relation. An asserted edge redundant with an
|
||||
# existing path is a future divergence.
|
||||
# MaterializedView a class standing in for an attribute; being
|
||||
# derived is what dependsOn already says.
|
||||
# approximateRowCount volumetry with no identified use. queryCount30d
|
||||
# and lastQueriedOn carry the rationalisation case.
|
||||
# hasVendor marginal.
|
||||
#
|
||||
# D. hasProductOwner drops from BLOCKING to a warning in the shapes. The SODH
|
||||
# product owner is genuinely not appointed yet, and a blocking rule that
|
||||
# cannot be satisfied truthfully forces invention -- the BR-004 lesson.
|
||||
#
|
||||
# Environment and deployedIn stay, unused. Without them a DEV table and its
|
||||
# PROD twin are indistinguishable and lineage pollutes silently on the first
|
||||
# multi-environment harvest. Unused because nothing is harvested yet, not
|
||||
# because they are useless.
|
||||
#
|
||||
# WHAT CHANGED vs v1.0 (kept for reference)
|
||||
# -----------------------------------------
|
||||
# A. RELATIONS SIMPLIFIED AND GIVEN HONEST NAMES
|
||||
# belongsToDomain + belongsToSubDomain -> belongsTo (one property,
|
||||
# scoped per class by SHACL; reusable if a domain family appears).
|
||||
# implements -> represents
|
||||
# composedOf -> aboutConcept (exactly 1, the
|
||||
# subject) + usesConcept (0..n, notions needed to read the object).
|
||||
# composedOf carried three different meanings depending on the object
|
||||
# (dependency, subject, part-whole); that vagueness is what let BR-004
|
||||
# be satisfied with padding in v0.6.
|
||||
#
|
||||
# B. represents WORKS AT TWO GRANULARITIES
|
||||
# DataObject represents BusinessObject
|
||||
# DataElement represents BusinessConcept
|
||||
# Same relation -- "this logical artefact is the data-side counterpart of
|
||||
# that business notion" -- at two levels of detail.
|
||||
# A Data Element reaches business meaning by EXACTLY ONE of two routes:
|
||||
# dimensional -> represents a BusinessConcept
|
||||
# measure -> is computedBy a Metric, which measures the Concept
|
||||
# Never both, never neither. Enforced by sh:xone.
|
||||
#
|
||||
# C. STEWARDSHIP DECLARED ONCE, INHERITED EVERYWHERE
|
||||
# monitoredBy sits on BusinessObject only (plus DataProduct, which
|
||||
# spans several Business Objects and cannot inherit). Data Objects and
|
||||
# Data Elements inherit through represents / hasElement. Declaring a
|
||||
# steward at several levels means two answers to "who watches this
|
||||
# column". BusinessObject therefore STAYS in the Ownership layer: it is
|
||||
# the single anchor of accountability, which is what that layer is for.
|
||||
# owningDomain is the exception: inherited by default, but re-declarable
|
||||
# on a Data Element, because OW-002 attributes an element to the domain
|
||||
# that defines its rule. A DE that re-declares is a borrowed element --
|
||||
# a signal to check against its represents, not an anomaly.
|
||||
#
|
||||
# D. BUSINESS MEANING LIVES IN ONE PLACE
|
||||
# hasBusinessDefinition is removed from BusinessObject. Meaning belongs to
|
||||
# the BusinessConcept; a Business Object gets its meaning through
|
||||
# aboutConcept. Two definitions for one notion are two definitions free to
|
||||
# diverge.
|
||||
#
|
||||
# E. GRAIN MOVED OFF THE METRIC
|
||||
# hasGrainDimension (Metric -> DataObject) and hasGranularity are gone.
|
||||
# Grain is a property of a DATASET, not of a calculation rule: a Metric
|
||||
# has no rows. hasGrainElement (DataObject -> DataElement) replaces them
|
||||
# and is CHECKABLE against the harvested PrimaryKey. A Metric's grain
|
||||
# becomes derivable from where it lands.
|
||||
#
|
||||
# F. DELIVERY IS ITS OWN LAYER
|
||||
# DataProduct / DataInterface / DataContract answer "what do we promise,
|
||||
# to whom, until when" -- a different question from "how is it
|
||||
# structured". DataAsset is renamed DataInterface and redefined as the
|
||||
# CHANNEL through which a consumer actually gets data (a Snowflake share
|
||||
# or grant, a file extract, an API), which is verifiable, rather than
|
||||
# "directly consumable by a business user", which was a judgement call and
|
||||
# produced zero instances in a year.
|
||||
# Every Data Product is operated by exactly one Sub-Domain and has exactly
|
||||
# one Product Owner: it is a consumable, and there must be no ambiguity
|
||||
# about who runs it.
|
||||
#
|
||||
# G. realizesMetric KEPT AS A TRANSITIONAL CONTROL
|
||||
# Declared, required on certified BI fields, and that is all. No rule is
|
||||
# written against the lineage chain: making the assertion conditional on
|
||||
# intermediate mappings existing would create confusion and discourage
|
||||
# people from filling it in at all.
|
||||
#
|
||||
# Deprecated terms are kept, never deleted (LC-002): existing assertions keep
|
||||
# classifying, and the migration can be staged.
|
||||
#
|
||||
# WHAT CHANGED vs v0.9.2 (kept for reference)
|
||||
# -------------------------------------------
|
||||
# 1. TERM NAMESPACE IS NOW VERSION-INDEPENDENT.
|
||||
# Terms live at .../metamodel/ (no version segment).
|
||||
# The VERSION lives in owl:versionIRI only.
|
||||
@@ -49,13 +151,13 @@
|
||||
|
||||
<https://ontology.pernod-ricard.com/metamodel/>
|
||||
a owl:Ontology ;
|
||||
owl:versionIRI <https://ontology.pernod-ricard.com/metamodel/1.0> ;
|
||||
owl:priorVersion <https://ontology.pernod-ricard.com/metamodel/0.9> ;
|
||||
owl:versionIRI <https://ontology.pernod-ricard.com/metamodel/1.2> ;
|
||||
owl:priorVersion <https://ontology.pernod-ricard.com/metamodel/1.1> ;
|
||||
dcterms:title "Pernod Ricard Data Meta Model" ;
|
||||
dcterms:creator "Bastien Gourdon" ;
|
||||
dcterms:modified "2026-07-25"^^xsd:date ;
|
||||
owl:versionInfo "1.0.0-draft-2026-07-25" ;
|
||||
rdfs:comment """OWL T-Box. Five layers (Ownership, Business, Logical, Physical, Consumption) crossed with a provenance axis (Curated vs Observed). Curated objects are asserted by governance and reviewed; Observed objects are harvested from Snowflake, Tableau and Looker and are rebuilt, never edited. Bridge annotation properties are declared for future alignment with domain ontologies.""" .
|
||||
dcterms:modified "2026-07-27"^^xsd:date ;
|
||||
owl:versionInfo "1.2.0-draft-2026-07-27" ;
|
||||
rdfs:comment """OWL T-Box. Six layers (Ownership, Business, Logical, Delivery, Physical, Consumption) crossed with a provenance axis (Curated vs Observed). Curated objects are asserted by governance and reviewed; Observed objects are harvested from Snowflake, Tableau and Looker and are rebuilt, never edited. Bridge annotation properties are declared for future alignment with domain ontologies.""" .
|
||||
|
||||
|
||||
#################################################################
|
||||
@@ -115,7 +217,11 @@ pr:OwnershipLayerObject a owl:Class ; rdfs:subClassOf pr:CuratedObject ;
|
||||
pr:BusinessLayerObject a owl:Class ; rdfs:subClassOf pr:CuratedObject ;
|
||||
rdfs:label "Business Layer Object" .
|
||||
pr:LogicalLayerObject a owl:Class ; rdfs:subClassOf pr:CuratedObject ;
|
||||
rdfs:label "Logical Layer Object" .
|
||||
rdfs:label "Logical Layer Object" ;
|
||||
rdfs:comment "How the data is structured." .
|
||||
pr:DeliveryLayerObject a owl:Class ; rdfs:subClassOf pr:CuratedObject ;
|
||||
rdfs:label "Delivery Layer Object" ;
|
||||
rdfs:comment "What we promise, to whom, until when. A different question from how the data is structured, hence a layer of its own: version, owner, changelog, support channel, consumers to warn." .
|
||||
pr:PhysicalLayerObject a owl:Class ; rdfs:subClassOf pr:ObservedObject ;
|
||||
rdfs:label "Physical Layer Object" .
|
||||
pr:ConsumptionLayerObject a owl:Class ; rdfs:subClassOf pr:ObservedObject ;
|
||||
@@ -135,7 +241,7 @@ pr:SubDomain a owl:Class ; rdfs:subClassOf pr:OwnershipLayerObject ;
|
||||
|
||||
pr:BusinessObject a owl:Class ; rdfs:subClassOf pr:OwnershipLayerObject ;
|
||||
rdfs:label "Business Object" ;
|
||||
rdfs:comment "The unit of ownership attribution. Sits in the Ownership layer rather than the Business layer on purpose: a Business Object is what a Data Domain is accountable for, and what a Data Object implements. Its business meaning is carried by the Business Concepts it groups." .
|
||||
rdfs:comment "The single anchor of accountability, which is why it stays in the Ownership layer: it is the one object where a steward is declared, and Data Objects and Data Elements inherit from it. It carries NO business definition of its own -- its meaning comes from its aboutConcept. What it does carry is a perimeter: the subject it is about, and the notions needed to read it." .
|
||||
|
||||
|
||||
#################################################################
|
||||
@@ -161,7 +267,7 @@ pr:KPI a owl:Class ; rdfs:subClassOf pr:Metric ;
|
||||
|
||||
pr:DataObject a owl:Class ; rdfs:subClassOf pr:LogicalLayerObject ;
|
||||
rdfs:label "Data Object" ;
|
||||
rdfs:comment "The pivot: implements exactly one Business Object, is composed of Data Elements, and is materialized as one or more Physical Relations. The single junction between governance and technology." .
|
||||
rdfs:comment "The pivot: represents exactly one Business Object, is composed of Data Elements, and is materialized as one or more Physical Relations. The single junction between governance and technology." .
|
||||
|
||||
pr:DataElement a owl:Class ; rdfs:subClassOf pr:LogicalLayerObject ;
|
||||
rdfs:label "Data Element" ;
|
||||
@@ -169,20 +275,31 @@ pr:DataElement a owl:Class ; rdfs:subClassOf pr:LogicalLayerObject ;
|
||||
rdfs:comment "The finest curated unit. Its mapping to a physical Field is the ONLY place where the curated graph and the observed graph meet: proposed automatically by name and type matching, confirmed by a Data Steward." .
|
||||
|
||||
pr:DataAsset a owl:Class ; rdfs:subClassOf pr:LogicalLayerObject ;
|
||||
rdfs:label "Data Asset" ;
|
||||
rdfs:comment "Directly consumable by a business user. A reusable building block is a Data Object, not a Data Asset (rule BR-009)." .
|
||||
|
||||
pr:DataProduct a owl:Class ; rdfs:subClassOf pr:LogicalLayerObject ;
|
||||
rdfs:label "Data Product" ;
|
||||
rdfs:comment "The unit of delivery: packages Data Assets and is governed by a Data Contract. Candidate alignment: dcat:Dataset / DPROD." .
|
||||
|
||||
pr:DataContract a owl:Class ; rdfs:subClassOf pr:LogicalLayerObject ;
|
||||
rdfs:label "Data Contract" ;
|
||||
rdfs:comment "The service commitment attached to a Data Product: schema, freshness, quality thresholds, support. Candidate alignment: ODCS (Bitol)." .
|
||||
rdfs:label "Data Asset (deprecated)" ;
|
||||
owl:deprecated true ;
|
||||
dcterms:isReplacedBy pr:DataInterface ;
|
||||
rdfs:comment "v1.0 term, defined as 'directly consumable by a business user' -- a judgement, not a structural test, which is why it collected zero instances in a year. Use DataInterface." .
|
||||
|
||||
|
||||
#################################################################
|
||||
# LAYER 4 - PHYSICAL (observed)
|
||||
# LAYER 4 - DELIVERY (curated)
|
||||
#################################################################
|
||||
|
||||
pr:DataProduct a owl:Class ; rdfs:subClassOf pr:DeliveryLayerObject ;
|
||||
rdfs:label "Data Product" ;
|
||||
rdfs:comment "The unit of delivery: what carries a version number, an owner, a changelog, a support channel and a list of consumers to warn. It has no single technical counterpart, and should not: physically it often lands on a Snowflake schema or a dbt project, but that is not what defines it. Operated by exactly one Sub-Domain -- it is a consumable, and there must be no ambiguity about who runs it. Candidate alignment: dcat:Dataset / DPROD." .
|
||||
|
||||
pr:DataInterface a owl:Class ; rdfs:subClassOf pr:DeliveryLayerObject ;
|
||||
rdfs:label "Data Interface" ;
|
||||
rdfs:comment "The channel through which a consumer actually gets the data: a Snowflake secure share or listing, a consumption schema with its grants, a file extract to a panel partner, an API or reverse-ETL feed. Not a Data Object (that is structure) and not a BI Data Source (that is one consumer among others). Verifiable: a grant, a share or an endpoint either exists or it does not. The output port in data-mesh terms." .
|
||||
|
||||
pr:DataContract a owl:Class ; rdfs:subClassOf pr:DeliveryLayerObject ;
|
||||
rdfs:label "Data Contract" ;
|
||||
rdfs:comment "The service commitment attached to a Data Product: schema, freshness, quality thresholds, support, breaking-change notice period. Candidate alignment: ODCS (Bitol)." .
|
||||
|
||||
|
||||
#################################################################
|
||||
# LAYER 5 - PHYSICAL (observed)
|
||||
#################################################################
|
||||
|
||||
pr:System a owl:Class ; rdfs:subClassOf pr:PhysicalLayerObject ;
|
||||
@@ -211,18 +328,14 @@ pr:View a owl:Class ; rdfs:subClassOf pr:PhysicalRelation ;
|
||||
rdfs:comment "Depends on other relations through dependsOn, so impact is transitive." .
|
||||
|
||||
pr:MaterializedView a owl:Class ; rdfs:subClassOf pr:PhysicalRelation ;
|
||||
rdfs:label "Materialized View" .
|
||||
rdfs:label "Materialized View (deprecated)" ;
|
||||
owl:deprecated true ; dcterms:isReplacedBy pr:View ;
|
||||
rdfs:comment "A class standing in for an attribute: what distinguishes it is being derived, which dependsOn already states." .
|
||||
|
||||
pr:ExternalTable a owl:Class ; rdfs:subClassOf pr:PhysicalRelation ;
|
||||
rdfs:label "External Table" ;
|
||||
rdfs:comment "Backed by files outside the warehouse. Marks the boundary where lineage leaves Snowflake." .
|
||||
|
||||
pr:Table a owl:Class ; rdfs:subClassOf pr:PhysicalRelation ;
|
||||
rdfs:label "Table (deprecated)" ;
|
||||
owl:deprecated true ;
|
||||
dcterms:isReplacedBy pr:BaseTable ;
|
||||
rdfs:comment "v0.9 term. Kept so that existing assertions still classify. Use BaseTable." .
|
||||
|
||||
pr:Field a owl:Class ; rdfs:subClassOf pr:PhysicalLayerObject ;
|
||||
rdfs:label "Field" ;
|
||||
pr:harvestSource "SNOWFLAKE.INFORMATION_SCHEMA.COLUMNS" ;
|
||||
@@ -246,7 +359,7 @@ pr:Transformation a owl:Class ; rdfs:subClassOf pr:PhysicalLayerObject , prov:Ac
|
||||
|
||||
|
||||
#################################################################
|
||||
# LAYER 5 - CONSUMPTION (observed)
|
||||
# LAYER 6 - CONSUMPTION (observed)
|
||||
#################################################################
|
||||
|
||||
pr:BIWorkspace a owl:Class ; rdfs:subClassOf pr:ConsumptionLayerObject ;
|
||||
@@ -276,9 +389,26 @@ pr:BIReport a owl:Class ; rdfs:subClassOf pr:ConsumptionLayerObject ;
|
||||
pr:Actor a owl:Class ; rdfs:label "Actor" ;
|
||||
rdfs:comment "A person or a team. Deliberately NOT a MetaModelObject: people are not subject to ownership and lifecycle rules." .
|
||||
|
||||
pr:DataDomainOwner a owl:Class ; rdfs:subClassOf pr:Actor ; rdfs:label "Data Domain Owner" .
|
||||
pr:DataGovernanceLead a owl:Class ; rdfs:subClassOf pr:Actor ; rdfs:label "Data Governance Lead (DGL)" .
|
||||
pr:DataSteward a owl:Class ; rdfs:subClassOf pr:Actor ; rdfs:label "Data Steward" .
|
||||
pr:DataDomainOwner a owl:Class ; rdfs:subClassOf pr:Actor ;
|
||||
rdfs:label "Data Domain Owner" ;
|
||||
rdfs:comment "Accountable for a Data Domain. Exactly one per domain." .
|
||||
|
||||
pr:SubDomainOwner a owl:Class ; rdfs:subClassOf pr:Actor ;
|
||||
rdfs:label "Sub-Domain Owner" ;
|
||||
rdfs:comment "Accountable for a Sub-Domain. Exactly one per sub-domain." .
|
||||
|
||||
pr:DataGovernanceLead a owl:Class ; rdfs:subClassOf pr:Actor ;
|
||||
rdfs:label "Data Governance Lead" ;
|
||||
pr:hasAcronym "DGL" ;
|
||||
rdfs:comment "Runs governance for a Data Domain. The acronym belongs in hasAcronym, not in the label (rule NR-005)." .
|
||||
|
||||
pr:DataProductOwner a owl:Class ; rdfs:subClassOf pr:Actor ;
|
||||
rdfs:label "Data Product Owner" ;
|
||||
rdfs:comment "Runs a Data Product day to day: releases, changelog, support, consumer communication. Belongs to the Sub-Domain that operates the product." .
|
||||
|
||||
pr:DataSteward a owl:Class ; rdfs:subClassOf pr:Actor ;
|
||||
rdfs:label "Data Steward" ;
|
||||
rdfs:comment "Monitors a Business Object. Declared there and nowhere else: Data Objects and Data Elements inherit their steward, so that no column can ever have two answers to 'who watches this'." .
|
||||
|
||||
|
||||
#################################################################
|
||||
@@ -308,23 +438,36 @@ pr:ExternalProvider a pr:SystemType ; rdfs:label "External Provider" .
|
||||
# OBJECT PROPERTIES - Ownership & Business
|
||||
#################################################################
|
||||
|
||||
pr:belongsTo a owl:ObjectProperty , owl:FunctionalProperty ;
|
||||
rdfs:label "belongs to" ;
|
||||
rdfs:comment "Position in the categorization tree. One property for the whole tree: SubDomain belongs to DataDomain, BusinessObject belongs to SubDomain. Scope is enforced per class by SHACL rather than by two near-identical properties, and the same relation will serve if a domain family ever appears. Deliberately NOT transitive: entailment would produce two values and break the maxCount 1 shape. To walk the ancestry, use the property path belongsTo+ in the query." .
|
||||
|
||||
pr:belongsToDomain a owl:ObjectProperty , owl:FunctionalProperty ;
|
||||
rdfs:label "belongs to domain" ;
|
||||
rdfs:domain pr:SubDomain ; rdfs:range pr:DataDomain .
|
||||
rdfs:label "belongs to domain (deprecated)" ;
|
||||
owl:deprecated true ; dcterms:isReplacedBy pr:belongsTo .
|
||||
|
||||
pr:belongsToSubDomain a owl:ObjectProperty , owl:FunctionalProperty ;
|
||||
rdfs:label "belongs to sub-domain" ;
|
||||
rdfs:domain pr:BusinessObject ; rdfs:range pr:SubDomain .
|
||||
rdfs:label "belongs to sub-domain (deprecated)" ;
|
||||
owl:deprecated true ; dcterms:isReplacedBy pr:belongsTo .
|
||||
|
||||
pr:aboutConcept a owl:ObjectProperty , owl:FunctionalProperty ;
|
||||
rdfs:label "is about" ;
|
||||
rdfs:domain pr:BusinessObject ; rdfs:range pr:BusinessConcept ;
|
||||
rdfs:comment "The single subject of this Business Object, and where its meaning comes from -- a Business Object carries no definition of its own. Exactly one. If the subject cannot be named, the object is not scoped, and that is the most useful thing the rule can tell you. Replaces the 3..10 minimum of BR-004, which was satisfied in v0.6 by attaching unrelated concepts." .
|
||||
|
||||
pr:usesConcept a owl:ObjectProperty ;
|
||||
rdfs:label "uses concept" ;
|
||||
rdfs:domain pr:BusinessObject ; rdfs:range pr:BusinessConcept ;
|
||||
rdfs:comment "The other notions required to read this object correctly. Zero or more, and zero is a legitimate answer." .
|
||||
|
||||
pr:composedOf a owl:ObjectProperty ;
|
||||
rdfs:label "composed of" ;
|
||||
rdfs:domain pr:BusinessObject ; rdfs:range pr:BusinessConcept ;
|
||||
owl:inverseOf pr:isComponentOf ;
|
||||
rdfs:comment "Part-whole. In v0.9 this was declared owl:inverseOf pr:describes, which was semantically wrong: composition and aboutness are different relations, and a reasoner would infer that every Concept describing an Object is also a component of it." .
|
||||
rdfs:label "composed of (deprecated)" ;
|
||||
owl:deprecated true ; dcterms:isReplacedBy pr:usesConcept ;
|
||||
rdfs:comment "Carried three different meanings depending on the object: dependency (Sell Out Volume), subject (Retailer Stock), part-whole (Calendar). One word for three relations is what allowed the v0.6 padding. Split into aboutConcept and usesConcept." .
|
||||
|
||||
pr:isComponentOf a owl:ObjectProperty ;
|
||||
rdfs:label "is component of" ;
|
||||
rdfs:domain pr:BusinessConcept ; rdfs:range pr:BusinessObject .
|
||||
rdfs:label "is component of (deprecated)" ;
|
||||
owl:deprecated true ; dcterms:isReplacedBy pr:usesConcept .
|
||||
|
||||
pr:describes a owl:ObjectProperty ;
|
||||
rdfs:label "describes (deprecated)" ;
|
||||
@@ -340,10 +483,15 @@ pr:measures a owl:ObjectProperty ;
|
||||
rdfs:label "measures" ;
|
||||
rdfs:domain pr:Metric ; rdfs:range pr:BusinessConcept .
|
||||
|
||||
pr:hasGrainElement a owl:ObjectProperty ;
|
||||
rdfs:label "has grain element" ;
|
||||
rdfs:domain pr:DataObject ; rdfs:range pr:DataElement ;
|
||||
rdfs:comment "The elements whose combination makes a row unique. Grain belongs to a DATASET, not to a calculation rule: a Metric has no rows, so it cannot have a grain of its own -- its grain is derivable from the Data Objects it lands in. Stated at element level rather than object level because it is then CHECKABLE against the harvested PrimaryKey: a declared grain that does not match the real key is an automatic finding." .
|
||||
|
||||
pr:hasGrainDimension a owl:ObjectProperty ;
|
||||
rdfs:label "has grain dimension" ;
|
||||
rdfs:domain pr:Metric ; rdfs:range pr:DataObject ;
|
||||
rdfs:comment "The dimensional Data Objects that define the Metric's grain. Replaces reading grain out of the free-text hasGranularity string: only a real reference lets you detect that two Metrics cannot be aggregated together." .
|
||||
rdfs:label "has grain dimension (deprecated)" ;
|
||||
owl:deprecated true ; dcterms:isReplacedBy pr:hasGrainElement ;
|
||||
rdfs:comment "Put grain on the Metric, which confounded a declarative statement with a structural fact. See hasGrainElement." .
|
||||
|
||||
pr:computedBy a owl:ObjectProperty ;
|
||||
rdfs:subPropertyOf prov:wasDerivedFrom ;
|
||||
@@ -356,10 +504,22 @@ pr:computedBy a owl:ObjectProperty ;
|
||||
# OBJECT PROPERTIES - Logical
|
||||
#################################################################
|
||||
|
||||
pr:represents a owl:ObjectProperty ;
|
||||
rdfs:label "represents" ;
|
||||
rdfs:comment """The business notion a logical artefact is the data-side counterpart of. One relation at two granularities:
|
||||
DataObject represents BusinessObject (exactly one -- the pivot, BR-006)
|
||||
DataElement represents BusinessConcept (at most one)
|
||||
Renamed from 'implements', which carried software-engineering baggage: a Data Object does not implement a Business Object the way a class implements an interface.
|
||||
|
||||
Range is deliberately left open and scoped by SHACL per class, because the two uses have different targets.
|
||||
|
||||
A Data Element uses this ONLY when it is dimensional. A measure element reaches meaning through Metric -> computedBy, which says more: it names WHICH metric the meaning travels through. Never both routes on one element.
|
||||
|
||||
Functional at DataObject level only -- not declared owl:FunctionalProperty here, since a single global cardinality would be wrong for one of the two uses.""" .
|
||||
|
||||
pr:implements a owl:ObjectProperty , owl:FunctionalProperty ;
|
||||
rdfs:label "implements" ;
|
||||
rdfs:domain pr:DataObject ; rdfs:range pr:BusinessObject ;
|
||||
rdfs:comment "The pivot. Exactly one, enforced by SHACL (BR-006)." .
|
||||
rdfs:label "implements (deprecated)" ;
|
||||
owl:deprecated true ; dcterms:isReplacedBy pr:represents .
|
||||
|
||||
pr:hasElement a owl:ObjectProperty ;
|
||||
rdfs:label "has element" ;
|
||||
@@ -370,18 +530,43 @@ pr:references a owl:ObjectProperty ;
|
||||
rdfs:domain pr:DataObject ; rdfs:range pr:DataObject ;
|
||||
rdfs:comment "Foreign-key reference between Data Objects. NOT ownership. Now checkable against the harvested ForeignKeyConstraint objects." .
|
||||
|
||||
pr:usesDataObject a owl:ObjectProperty ;
|
||||
rdfs:label "uses Data Object" ;
|
||||
rdfs:domain pr:DataAsset ; rdfs:range pr:DataObject .
|
||||
#################################################################
|
||||
# OBJECT PROPERTIES - Delivery
|
||||
#################################################################
|
||||
|
||||
pr:packages a owl:ObjectProperty ;
|
||||
rdfs:label "packages" ;
|
||||
rdfs:domain pr:DataProduct ; rdfs:range pr:DataAsset .
|
||||
rdfs:domain pr:DataProduct ; rdfs:range pr:DataObject ;
|
||||
rdfs:comment "The scope of the product. Through these Data Objects and their represents, the Business Objects a product makes available are derivable -- which is why a product needs no direct link to a Business Object." .
|
||||
|
||||
pr:governedBy a owl:ObjectProperty ;
|
||||
pr:exposes a owl:ObjectProperty ;
|
||||
rdfs:label "exposes" ;
|
||||
rdfs:domain pr:DataProduct ; rdfs:range pr:DataInterface ;
|
||||
rdfs:comment "The channels through which the product is actually consumed." .
|
||||
|
||||
pr:servedBy a owl:ObjectProperty ;
|
||||
rdfs:label "served by" ;
|
||||
rdfs:domain pr:DataInterface ; rdfs:range pr:PhysicalRelation ;
|
||||
rdfs:comment "What physically backs the interface: the shared relations, the extract source." .
|
||||
|
||||
pr:governedBy a owl:ObjectProperty , owl:FunctionalProperty ;
|
||||
rdfs:label "governed by" ;
|
||||
rdfs:domain pr:DataProduct ; rdfs:range pr:DataContract .
|
||||
|
||||
pr:operatedBy a owl:ObjectProperty , owl:FunctionalProperty ;
|
||||
rdfs:label "operated by" ;
|
||||
rdfs:domain pr:DataProduct ; rdfs:range pr:SubDomain ;
|
||||
rdfs:comment "The single Sub-Domain that runs this product. A Data Product spans several Business Objects and therefore cannot inherit accountability the way a Data Object does -- so it declares its own, and exactly one. It is a consumable: there must be no ambiguity about who runs it." .
|
||||
|
||||
pr:hasProductOwner a owl:ObjectProperty , owl:FunctionalProperty ;
|
||||
rdfs:label "has product owner" ;
|
||||
rdfs:domain pr:DataProduct ; rdfs:range pr:DataProductOwner ;
|
||||
rdfs:comment "Exactly one. Belongs to the operating Sub-Domain -- checkable, not just conventional." .
|
||||
|
||||
pr:usesDataObject a owl:ObjectProperty ;
|
||||
rdfs:label "uses Data Object (deprecated)" ;
|
||||
owl:deprecated true ; dcterms:isReplacedBy pr:packages .
|
||||
|
||||
|
||||
#################################################################
|
||||
# OBJECT PROPERTIES - the curated / observed seam
|
||||
@@ -395,11 +580,15 @@ pr:materializedAs a owl:ObjectProperty ;
|
||||
pr:curationMode "PROPOSED_CONFIRMED" ;
|
||||
rdfs:comment "Data Object to physical relation. Not functional: the same Data Object exists in DEV, UAT and PROD, and sometimes across several tables." .
|
||||
|
||||
pr:physicalizedIn a owl:ObjectProperty ;
|
||||
rdfs:label "physicalized in" ;
|
||||
pr:storedIn a owl:ObjectProperty ;
|
||||
rdfs:label "stored in" ;
|
||||
rdfs:domain pr:DataElement ; rdfs:range pr:Field ;
|
||||
pr:curationMode "PROPOSED_CONFIRMED" ;
|
||||
rdfs:comment "Data Element to physical column, 1..n. Proposed by the reconciliation job on fullyQualifiedName and type, confirmed by a Data Steward. This single edge is what makes end-to-end impact analysis possible." .
|
||||
rdfs:comment "Data Element to physical column, 1..n. Proposed by the reconciliation job on fullyQualifiedName and type, confirmed by a Data Steward. This single edge is what makes end-to-end impact analysis possible. Renamed from physicalizedIn, which was jargon: the relation says where the data sits." .
|
||||
|
||||
pr:physicalizedIn a owl:ObjectProperty ;
|
||||
rdfs:label "physicalized in (deprecated)" ;
|
||||
owl:deprecated true ; dcterms:isReplacedBy pr:storedIn .
|
||||
|
||||
|
||||
#################################################################
|
||||
@@ -442,8 +631,9 @@ pr:deployedIn a owl:ObjectProperty , owl:FunctionalProperty ;
|
||||
rdfs:domain pr:ObservedObject ; rdfs:range pr:Environment .
|
||||
|
||||
pr:constrainsRelation a owl:ObjectProperty , owl:FunctionalProperty ;
|
||||
rdfs:label "constrains relation" ;
|
||||
rdfs:domain pr:KeyConstraint ; rdfs:range pr:PhysicalRelation .
|
||||
rdfs:label "constrains relation (deprecated)" ;
|
||||
owl:deprecated true ;
|
||||
rdfs:comment "Derivable: constrainsField then livesIn gives the relation. An asserted edge redundant with an existing path is a divergence waiting to happen." .
|
||||
|
||||
pr:constrainsField a owl:ObjectProperty ;
|
||||
rdfs:label "constrains field" ;
|
||||
@@ -513,7 +703,13 @@ pr:realizesMetric a owl:ObjectProperty , owl:FunctionalProperty ;
|
||||
rdfs:label "realizes metric" ;
|
||||
rdfs:domain pr:BIField ; rdfs:range pr:Metric ;
|
||||
pr:curationMode "PROPOSED_CONFIRMED" ;
|
||||
rdfs:comment "The harmonisation proof. Counting distinct BI Fields realizing one certified Metric, and comparing their expressions, turns 'we harmonised the definitions' from a claim into a measurement over the real BI estate." .
|
||||
rdfs:comment """The harmonisation measure: how many distinct BI Fields realize one certified Metric, and do their expressions agree. Turns 'we harmonised the definitions' from a claim into a count over the real BI estate.
|
||||
|
||||
In principle this is derivable from the lineage chain Metric -> DataElement -> Field -> BIField, since lineage carries semantic inheritance. In practice that chain will have holes -- four hops, one of them PROPOSED_CONFIRMED, another harvested from an API that does not resolve custom SQL, extracts or cascaded calculated fields -- and the question it answers is too important to depend on them.
|
||||
|
||||
So: TRANSITIONAL CONTROL. Asserted directly, and that is all. No rule is written against the lineage chain: making the assertion conditional on intermediate mappings would create confusion and discourage people from filling it in.
|
||||
|
||||
Retirement criterion, to be measured rather than debated: when the lineage chain reproduces the asserted link for more than 95% of certified BI fields across three consecutive reconciliation cycles, the direct link is derivable and can go.""" .
|
||||
|
||||
pr:usesBIDataSource a owl:ObjectProperty ;
|
||||
rdfs:label "uses BI data source" ;
|
||||
@@ -534,15 +730,34 @@ pr:ownedBy a owl:ObjectProperty ;
|
||||
|
||||
pr:owningDomain a owl:ObjectProperty , owl:FunctionalProperty ;
|
||||
rdfs:label "owning Data Domain" ; rdfs:range pr:DataDomain ;
|
||||
rdfs:comment "The single Data Domain with authority to define the object's rule. Required on Curated Objects, absent on Observed Objects: a Snowflake column is attributed through the Data Element mapped to it, never asserted directly." .
|
||||
rdfs:comment """The single Data Domain with authority to define the object's rule. Absent on Observed Objects: a Snowflake column is attributed through the Data Element mapped to it, never asserted directly.
|
||||
|
||||
Unlike the steward, this one is RE-DECLARABLE on a Data Element. It is inherited by default from the parent chain, but OW-002 attributes an element to the domain that defines its RULE, and a borrowed element -- Brand Code sitting in a Sell Out fact object -- has its rule elsewhere. A Data Element whose owningDomain differs from its parent's is a borrowed element: a signal to check against its represents, not an anomaly to fix.""" .
|
||||
|
||||
pr:monitoredBy a owl:ObjectProperty ;
|
||||
rdfs:label "monitored by" ; rdfs:range pr:DataSteward ;
|
||||
rdfs:comment """Declared on BusinessObject, and on DataProduct, and NOWHERE ELSE. Data Objects inherit through represents, Data Elements through hasElement. Declaring a steward at several levels would allow two answers to 'who watches this column'.
|
||||
|
||||
Says what the steward DOES, where the deprecated hasSteward only stated a membership.
|
||||
|
||||
DataProduct is the one exception to inheritance: it packages Data Objects belonging to several Business Objects, so it cannot inherit a single steward and declares its own.""" .
|
||||
|
||||
pr:hasSteward a owl:ObjectProperty ;
|
||||
rdfs:label "has Data Steward" ; rdfs:range pr:DataSteward .
|
||||
rdfs:label "has Data Steward (deprecated)" ;
|
||||
owl:deprecated true ; dcterms:isReplacedBy pr:monitoredBy .
|
||||
|
||||
pr:hasDGL a owl:ObjectProperty , owl:FunctionalProperty ;
|
||||
rdfs:label "has DGL" ;
|
||||
rdfs:domain pr:DataDomain ; rdfs:range pr:DataGovernanceLead .
|
||||
|
||||
pr:hasDomainOwner a owl:ObjectProperty , owl:FunctionalProperty ;
|
||||
rdfs:label "has domain owner" ;
|
||||
rdfs:domain pr:DataDomain ; rdfs:range pr:DataDomainOwner .
|
||||
|
||||
pr:hasSubDomainOwner a owl:ObjectProperty , owl:FunctionalProperty ;
|
||||
rdfs:label "has sub-domain owner" ;
|
||||
rdfs:domain pr:SubDomain ; rdfs:range pr:SubDomainOwner .
|
||||
|
||||
pr:hasActivationStatus a owl:ObjectProperty , owl:FunctionalProperty ;
|
||||
rdfs:label "activation status" ;
|
||||
rdfs:domain pr:DataDomain ; rdfs:range pr:ActivationStatus .
|
||||
@@ -558,12 +773,24 @@ pr:hasStatus a owl:DatatypeProperty ; rdfs:label "status" ; rdfs:range xsd:s
|
||||
pr:hasVersion a owl:DatatypeProperty ; rdfs:label "version" ; rdfs:range xsd:string .
|
||||
pr:createdOn a owl:DatatypeProperty ; rdfs:label "created on" ; rdfs:range xsd:date .
|
||||
pr:lastReviewedOn a owl:DatatypeProperty ; rdfs:label "last reviewed on" ; rdfs:range xsd:date .
|
||||
pr:hasSource a owl:DatatypeProperty ; rdfs:label "back-doc source" ; rdfs:range xsd:string .
|
||||
pr:hasSource a owl:DatatypeProperty ; rdfs:label "source of truth" ; rdfs:range xsd:string ;
|
||||
rdfs:comment """The system holding the authoritative data for this object: SODH Gold, MDM Product, Group Finance.
|
||||
|
||||
ONE MEANING ONLY. It had drifted into naming the document a record came from ("SODH back-doc v0.7"), and into packing a system and a relation into one string ("SODH Gold / F_SELL_OUT"). The relation belongs in physicalName; the document belongs nowhere -- provenance of a record is not a property of the thing it describes.
|
||||
|
||||
MUST NOT appear on a Business Concept: a concept has no data source. Its authority is its owning domain and its definition is published in the glossary. Enforced by SHACL.
|
||||
|
||||
On a Metric it is asserted rather than derived, by exception, because "where can I get this metric" is a first-order question and the derivation chain (computedBy, then the data object holding the element) will have gaps before harvesting. A shape checks the assertion against the elements.""" .
|
||||
pr:arbitrationStatus a owl:DatatypeProperty ; rdfs:label "arbitration status" ; rdfs:range xsd:string ;
|
||||
rdfs:comment "TO_ARBITRATE blocks publication (OW-007). Use this on the object rather than tracking arbitration in a side spreadsheet." .
|
||||
|
||||
# ---- Business semantics
|
||||
pr:hasBusinessDefinition a owl:DatatypeProperty ; rdfs:label "business definition" ; rdfs:range xsd:string .
|
||||
pr:hasBusinessDefinition a owl:DatatypeProperty ; rdfs:label "business definition" ; rdfs:range xsd:string ;
|
||||
rdfs:comment """The meaning of a business notion. Belongs to the BusinessConcept.
|
||||
|
||||
MUST NOT be carried by a BusinessObject: an object gets its meaning through aboutConcept. Two definitions for one notion are two definitions free to diverge. Enforced by SHACL rather than by rdfs:domain, because Data Elements legitimately carry one too and a declared domain would make them entail BusinessConcept -- which the disjointness axioms would then reject.
|
||||
|
||||
MIGRATION: text currently on a Business Object moves to the Concept named by its aboutConcept.""" .
|
||||
pr:hasTechnicalDefinition a owl:DatatypeProperty ; rdfs:label "technical definition" ; rdfs:range xsd:string .
|
||||
pr:hasAcronym a owl:DatatypeProperty ; rdfs:label "acronym" ; rdfs:range xsd:string .
|
||||
pr:hasSynonym a owl:DatatypeProperty ; rdfs:label "synonym" ; rdfs:range xsd:string .
|
||||
@@ -573,8 +800,9 @@ pr:hasBusinessRule a owl:DatatypeProperty ; rdfs:label "business rule" ; rdfs:ra
|
||||
pr:hasFormula a owl:DatatypeProperty ; rdfs:label "formula" ; rdfs:range xsd:string .
|
||||
pr:hasUnit a owl:DatatypeProperty ; rdfs:label "unit" ; rdfs:range xsd:string ;
|
||||
rdfs:comment "From the Units LoV. Note the LoV currently mixes physical units, currencies (ISO 4217) and value types (Index, Rate, Count) - three different concepts that should eventually be split." .
|
||||
pr:hasGranularity a owl:DatatypeProperty ; rdfs:label "granularity (label)" ; rdfs:range xsd:string ;
|
||||
rdfs:comment "Human-readable grain label, e.g. 'SKU x Customer x Day'. Keep in sync with hasGrainDimension, which is the machine-readable form." .
|
||||
pr:hasGranularity a owl:DatatypeProperty ; rdfs:label "granularity (deprecated)" ; rdfs:range xsd:string ;
|
||||
owl:deprecated true ; dcterms:isReplacedBy pr:hasGrainElement ;
|
||||
rdfs:comment "Free-text grain label on a Metric, e.g. 'SKU x Customer x Day'. Kept a second, unverifiable copy of something now stated structurally on the Data Object. Grain is a property of a dataset, not of a calculation rule." .
|
||||
pr:hasTimeAggregation a owl:DatatypeProperty ; rdfs:label "time aggregation" ; rdfs:range xsd:string .
|
||||
|
||||
pr:isKPI a owl:DatatypeProperty ; rdfs:label "is KPI (deprecated)" ; rdfs:range xsd:boolean ;
|
||||
@@ -612,12 +840,15 @@ pr:hasPhysicalDataType a owl:DatatypeProperty ; rdfs:label "physical data type"
|
||||
pr:isNullable a owl:DatatypeProperty ; rdfs:label "is nullable" ; rdfs:range xsd:boolean .
|
||||
pr:ordinalPosition a owl:DatatypeProperty ; rdfs:label "ordinal position" ; rdfs:range xsd:integer .
|
||||
pr:hasViewDefinition a owl:DatatypeProperty ; rdfs:label "view definition" ; rdfs:range xsd:string .
|
||||
pr:hasVendor a owl:DatatypeProperty ; rdfs:label "vendor" ; rdfs:range xsd:string .
|
||||
pr:hasVendor a owl:DatatypeProperty ; rdfs:label "vendor (deprecated)" ; rdfs:range xsd:string ;
|
||||
owl:deprecated true .
|
||||
pr:definedBy a owl:DatatypeProperty ; rdfs:label "defined by (code URI)" ; rdfs:range xsd:anyURI ;
|
||||
rdfs:comment "Git URI of the code behind a Transformation. Connects the graph to the repository." .
|
||||
|
||||
# ---- Usage evidence (the rationalisation argument)
|
||||
pr:approximateRowCount a owl:DatatypeProperty ; rdfs:label "approximate row count" ; rdfs:range xsd:integer .
|
||||
pr:approximateRowCount a owl:DatatypeProperty ; rdfs:label "approximate row count (deprecated)" ; rdfs:range xsd:integer ;
|
||||
owl:deprecated true ;
|
||||
rdfs:comment "Volumetry with no identified use. queryCount30d and lastQueriedOn carry the rationalisation argument; row count does not." .
|
||||
pr:lastQueriedOn a owl:DatatypeProperty ; rdfs:label "last queried on" ; rdfs:range xsd:dateTime ;
|
||||
pr:harvestSource "SNOWFLAKE.ACCOUNT_USAGE.ACCESS_HISTORY" ;
|
||||
rdfs:comment "Lets the model say 'documented but unused for eight months'. A rationalisation argument that costs nothing and is denominated in warehouse spend." .
|
||||
@@ -636,10 +867,10 @@ pr:hasExpression a owl:DatatypeProperty ; rdfs:label "BI expression" ; rdfs:rang
|
||||
|
||||
[] a owl:AllDisjointClasses ; owl:members (
|
||||
pr:OwnershipLayerObject pr:BusinessLayerObject pr:LogicalLayerObject
|
||||
pr:PhysicalLayerObject pr:ConsumptionLayerObject ) .
|
||||
pr:DeliveryLayerObject pr:PhysicalLayerObject pr:ConsumptionLayerObject ) .
|
||||
|
||||
[] a owl:AllDisjointClasses ; owl:members (
|
||||
pr:BusinessObject pr:DataObject pr:DataAsset pr:DataProduct ) .
|
||||
pr:BusinessObject pr:DataObject pr:DataInterface pr:DataProduct ) .
|
||||
|
||||
[] a owl:AllDisjointClasses ; owl:members ( pr:BusinessConcept pr:Metric ) .
|
||||
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
# PR Data Meta Model - validation stack
|
||||
#
|
||||
# PINNED ON PURPOSE. The venv on GrosseBertha runs Python 3.9
|
||||
# (/usr/local/bin/python3.9, the only stable interpreter on DSM 7.4).
|
||||
#
|
||||
# Newer pyshacl releases advertise 3.9 support but use PEP 604 syntax
|
||||
# (str | X) in runtime annotations, which needs 3.10+. pip installs them
|
||||
# without complaint and the import then dies with:
|
||||
# TypeError: unsupported operand type(s) for |: 'type' and 'ABCMeta'
|
||||
#
|
||||
# 0.26.0 is the version verified working on this interpreter (2026-07-26).
|
||||
# Do not unpin without rebuilding the venv and re-running the validator.
|
||||
#
|
||||
# /usr/local/bin/python3.9 -m venv venv
|
||||
# . venv/bin/activate && pip install -r requirements.txt
|
||||
|
||||
pyshacl==0.26.0
|
||||
rdflib>=6.3.2,<7.1
|
||||
@@ -0,0 +1,279 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
SODH arbitration - composedOf -> aboutConcept + usesConcept
|
||||
============================================================
|
||||
Second half of the v1.0 -> v1.1 migration. migrate_instances_v1_1.py did the
|
||||
mechanical renames and deliberately stopped here, because everything below is a
|
||||
judgement someone had to make.
|
||||
|
||||
USAGE
|
||||
python3 scripts/apply_arbitration_sodh.py # dry run
|
||||
python3 scripts/apply_arbitration_sodh.py --apply # rewrite, .bak kept
|
||||
|
||||
WHAT WAS ARBITRATED (Bastien, 2026-07-27)
|
||||
-----------------------------------------
|
||||
The v0.6 padding is what made this necessary: four Business Objects belonging
|
||||
to other domains -- Product Structure, Customer & Outlet, Currency, Calendar --
|
||||
were all declared composedOf the same three Sell Out concepts, because BR-004
|
||||
demanded three and there were none to give. The subject rule replaces that
|
||||
minimum: exactly one subject, and if it cannot be named the object is not
|
||||
scoped.
|
||||
|
||||
1. Distribution -> Retail Distribution.
|
||||
"Distribution" is too generic to survive contact with other domains, and
|
||||
it is NOT a kind of sell out: sell out counts units sold, distribution
|
||||
measures product presence in store. Retail Distribution follows panel
|
||||
vocabulary (Nielsen/Circana), under which Numeric and Weighted
|
||||
Distribution are the metrics.
|
||||
|
||||
2. Sell Out Baseline becomes a concept in its own right, not a flavour of
|
||||
Sell Out: modelled non-promoted volume is a different notion from
|
||||
observed volume.
|
||||
|
||||
3. Customer & Outlet splits into two Business Objects, and Currency &
|
||||
Exchange Rates likewise. An object named "X & Y" has no single subject,
|
||||
which is precisely what the rule is meant to surface. The split is
|
||||
PROPOSED here; DD-04 and DD-16 own the decision.
|
||||
|
||||
4. Six concepts belonging to other domains are created with a DRAFT
|
||||
definition and arbitrationStatus TO_ARBITRATE. A draft to correct beats a
|
||||
blank to fill, and OW-007 blocks publication until the owning domain has
|
||||
ratified. These definitions are the DGO's proposal, not DD-04/10/16/21's
|
||||
word.
|
||||
|
||||
NOT DONE HERE
|
||||
hasGranularity on Metrics still needs re-expressing as hasGrainElement on
|
||||
Data Objects, and the BR-013 alignment (10 mother metrics, computedBy)
|
||||
is a separate migration. Until that lands, every Data Element will fail
|
||||
the XOR-of-meaning shape -- which is the correct, informative result.
|
||||
"""
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
TARGET = os.path.join(REPO, "instances", "sodh.ttl")
|
||||
W = 78
|
||||
|
||||
# ---------------------------------------------------------------- new concepts
|
||||
# (iri, identifier, name, domain, steward, definition, arbitrate)
|
||||
NEW_CONCEPTS = [
|
||||
("ex:BC_06_01_005", "BC-06.01-005", "Sell Out Baseline", "ex:DD_06", "ex:ST_DD_06",
|
||||
"The modelled level of retail sales that would have occurred without promotional activity, "
|
||||
"used as the reference against which promotional uplift is measured. Distinct from Sell Out, "
|
||||
"which is observed rather than modelled.",
|
||||
False),
|
||||
("ex:BC_10_01_001", "BC-10.01-001", "Product", "ex:DD_10", "ex:ST_DD_10",
|
||||
"A commercial item sold by Pernod Ricard, identified at Stock Keeping Unit level and "
|
||||
"positioned in the corporate hierarchy from Brand through Category, Sub-Category and Flavor.",
|
||||
True),
|
||||
("ex:BC_04_01_001", "BC-04.01-001", "Customer", "ex:DD_04", "ex:ST_DD_04",
|
||||
"A commercial counterparty that buys Pernod Ricard products for resale, positioned in the "
|
||||
"trade hierarchy from Distributor through Tier-1 and Tier-2 Customer. Distinct from Outlet, "
|
||||
"which is a physical point of sale.",
|
||||
True),
|
||||
("ex:BC_04_01_002", "BC-04.01-002", "Outlet", "ex:DD_04", "ex:ST_DD_04",
|
||||
"A physical point of sale where products are made available to end consumers, characterised "
|
||||
"by its Channel. An Outlet belongs to a Customer but is not the same notion: one Customer "
|
||||
"operates many Outlets.",
|
||||
True),
|
||||
("ex:BC_16_01_001", "BC-16.01-001", "Currency", "ex:DD_16", "ex:ST_DD_16",
|
||||
"A monetary unit in which a transaction or a reported amount is denominated, identified by "
|
||||
"its ISO 4217 code.",
|
||||
True),
|
||||
("ex:BC_16_01_002", "BC-16.01-002", "Exchange Rate", "ex:DD_16", "ex:ST_DD_16",
|
||||
"The conversion factor applied between two currencies at a given date, used to restate local "
|
||||
"amounts into group reporting currency. Distinct from Currency: a rate is a dated "
|
||||
"relationship between two of them.",
|
||||
True),
|
||||
("ex:BC_21_01_001", "BC-21.01-001", "Calendar Date", "ex:DD_21", "ex:ST_DD_21",
|
||||
"A single day in the Gregorian calendar, the atomic unit from which Week, Month, Quarter and "
|
||||
"Year are derived and against which all time-based measures are positioned.",
|
||||
True),
|
||||
]
|
||||
|
||||
# ------------------------------------------------------------------- new object
|
||||
NEW_OBJECTS = [
|
||||
("ex:BO_04_02_001", "BO-04.02-001", "Outlet", "ex:DD_04", "ex:ST_DD_04", "ex:SD_04_02",
|
||||
"ex:BC_04_01_002", [], "MDM Customer -> SODH [split from BO-04.01-001, proposed to DD-04]"),
|
||||
("ex:BO_16_01_002", "BO-16.01-002", "Exchange Rate", "ex:DD_16", "ex:ST_DD_16", "ex:SD_16_01",
|
||||
"ex:BC_16_01_002", [], "Group Finance -> SODH [split from BO-16.01-001, proposed to DD-16]"),
|
||||
]
|
||||
|
||||
# ---------------------------------------------------- subject / used per object
|
||||
# iri -> (aboutConcept, [usesConcept...], new name or None)
|
||||
ARBITRATION = {
|
||||
"ex:BO_06_01_001": ("ex:BC_06_01_001", ["ex:BC_06_01_003", "ex:BC_05_01_001"], None),
|
||||
"ex:BO_06_01_002": ("ex:BC_06_01_001", ["ex:BC_06_01_003", "ex:BC_05_01_001",
|
||||
"ex:BC_16_01_001"], None),
|
||||
"ex:BO_06_01_003": ("ex:BC_06_01_004", ["ex:BC_06_01_003"], None),
|
||||
"ex:BO_06_01_004": ("ex:BC_06_01_002", ["ex:BC_06_01_001", "ex:BC_06_01_003"], None),
|
||||
"ex:BO_06_01_005": ("ex:BC_06_01_005", ["ex:BC_06_01_001", "ex:BC_05_01_001"], None),
|
||||
"ex:BO_10_01_001": ("ex:BC_10_01_001", [], None),
|
||||
"ex:BO_04_01_001": ("ex:BC_04_01_001", [], "Customer"),
|
||||
"ex:BO_16_01_001": ("ex:BC_16_01_001", [], "Currency"),
|
||||
"ex:BO_21_01_001": ("ex:BC_21_01_001", [], None),
|
||||
"ex:BO_05_01_001": ("ex:BC_05_01_001", [], None),
|
||||
}
|
||||
|
||||
RENAME_CONCEPT = {"ex:BC_06_01_004": ("Distribution", "Retail Distribution")}
|
||||
|
||||
|
||||
def concept_block(iri, ident, name, domain, steward, definition, arbitrate):
|
||||
lines = [
|
||||
'%s a pr:BusinessConcept ;' % iri,
|
||||
' pr:hasIdentifier "%s" ; pr:hasName "%s" ;' % (ident, name),
|
||||
' pr:owningDomain %s ; pr:ownedBy %s ;' % (domain, steward),
|
||||
' pr:hasBusinessDefinition "%s" ;' % definition,
|
||||
]
|
||||
if arbitrate:
|
||||
lines.append(' pr:arbitrationStatus "TO_ARBITRATE" ;')
|
||||
src = "DGO proposal, pending ratification by the owning domain"
|
||||
else:
|
||||
src = "SODH Gold"
|
||||
lines.append(' pr:hasStatus "DRAFT" ; pr:hasVersion "1.1" ; pr:hasSource "%s" .' % src)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def object_block(iri, ident, name, domain, steward, subdomain, about, uses, source):
|
||||
lines = [
|
||||
'%s a pr:BusinessObject ;' % iri,
|
||||
' pr:hasIdentifier "%s" ; pr:hasName "%s" ;' % (ident, name),
|
||||
' pr:owningDomain %s ; pr:ownedBy %s ;' % (domain, steward),
|
||||
' pr:belongsTo %s ; pr:monitoredBy %s ;' % (subdomain, steward),
|
||||
' pr:aboutConcept %s ;' % about,
|
||||
]
|
||||
if uses:
|
||||
lines.append(' pr:usesConcept %s ;' % " , ".join(uses))
|
||||
lines.append(' pr:arbitrationStatus "TO_ARBITRATE" ;')
|
||||
lines.append(' pr:hasStatus "DRAFT" ; pr:hasVersion "1.1" ; pr:hasSource "%s" .' % source)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main():
|
||||
apply_changes = "--apply" in sys.argv
|
||||
if not os.path.exists(TARGET):
|
||||
print("Not found: %s" % TARGET)
|
||||
sys.exit(2)
|
||||
|
||||
text = original = open(TARGET, encoding="utf-8").read()
|
||||
report = []
|
||||
|
||||
# ---- 1. rename the over-generic concept ----------------------------------
|
||||
for iri, (old, new) in RENAME_CONCEPT.items():
|
||||
pat = re.compile(r'(%s\b.*?pr:hasName )"%s"' % (re.escape(iri), re.escape(old)), re.S)
|
||||
text, n = pat.subn(r'\1"%s"' % new, text, count=1)
|
||||
if n:
|
||||
report.append("renamed concept %s: %s -> %s" % (iri, old, new))
|
||||
|
||||
# ---- 2. rewrite each Business Object -------------------------------------
|
||||
for iri, (about, uses, newname) in ARBITRATION.items():
|
||||
m = re.search(r'(^%s a pr:BusinessObject ;.*?\n)(?=^ex:|\Z)' % re.escape(iri),
|
||||
text, re.S | re.M)
|
||||
if not m:
|
||||
report.append("!! block not found: %s" % iri)
|
||||
continue
|
||||
block = m.group(1)
|
||||
|
||||
# strip what a previous run may have written, so re-running is a no-op.
|
||||
# Duplicated clauses carrying the SAME value merge under RDF set
|
||||
# semantics: the graph stays valid and SHACL sees nothing, while the
|
||||
# file quietly grows a redundant line on every pass.
|
||||
new_block = re.sub(r'[ \t]*pr:composedOf\s+[^";]*;[ \t]*\n', '', block)
|
||||
new_block = re.sub(r'[ \t]*pr:(aboutConcept|usesConcept)\s+[^";]*;[ \t]*\n',
|
||||
'', new_block)
|
||||
clause = ' pr:aboutConcept %s ;\n' % about
|
||||
if uses:
|
||||
clause += ' pr:usesConcept %s ;\n' % " , ".join(uses)
|
||||
new_block = re.sub(r'(\n)([ \t]*pr:hasMetric|[ \t]*pr:hasBusinessDefinition|[ \t]*pr:hasStatus)',
|
||||
r'\1%s\2' % clause, new_block, count=1)
|
||||
|
||||
# meaning belongs to the Concept, never to the Object
|
||||
# match the full quoted literal: several definitions contain a semicolon
|
||||
# inside the string ("Owned by Product & Material; consumed by SODH."),
|
||||
# which a naive [^;]* would stop at, silently leaving the property behind
|
||||
new_block, ndef = re.subn(
|
||||
r'[ \t]*pr:hasBusinessDefinition\s+"(?:[^"\\]|\\.)*"\s*;[ \t]*\n',
|
||||
'', new_block)
|
||||
|
||||
if newname:
|
||||
new_block = re.sub(r'(pr:hasName )"[^"]*"', r'\1"%s"' % newname, new_block, count=1)
|
||||
new_block = re.sub(r'(pr:hasSource )"[^"]*"',
|
||||
r'\1"MDM -> SODH [split proposed, subject arbitrated 2026-07-27]"',
|
||||
new_block, count=1)
|
||||
|
||||
text = text[:m.start(1)] + new_block + text[m.end(1):]
|
||||
report.append("%-18s about=%s uses=%d%s%s"
|
||||
% (iri, about.replace("ex:BC_", "BC-"), len(uses),
|
||||
" def->concept" if ndef else "",
|
||||
" renamed=%s" % newname if newname else ""))
|
||||
|
||||
# ---- 3. insert the missing concepts --------------------------------------
|
||||
# idempotent: identical duplicated blocks merge in RDF, so SHACL would
|
||||
# never flag them -- the file grows while the graph stays valid.
|
||||
todo_c = [c for c in NEW_CONCEPTS
|
||||
if not re.search(r'^%s a ' % re.escape(c[0]), text, re.M)]
|
||||
anchor = re.search(r'\n(?=ex:BO_\w+ a pr:BusinessObject ;)', text)
|
||||
if anchor and todo_c:
|
||||
blocks = "\n".join(concept_block(*c) for c in todo_c)
|
||||
text = (text[:anchor.start()] + "\n\n" +
|
||||
"# --- concepts added in v1.1 migration -------------------------------\n" +
|
||||
"# Six of these belong to other domains: DRAFT definitions proposed by the\n" +
|
||||
"# DGO, flagged TO_ARBITRATE. OW-007 blocks publication until ratified.\n" +
|
||||
blocks + "\n" + text[anchor.start():])
|
||||
report.append("inserted %d Business Concepts (%d TO_ARBITRATE)"
|
||||
% (len(todo_c), sum(1 for c in todo_c if c[6])))
|
||||
elif not todo_c:
|
||||
report.append("all %d Business Concepts already present, skipped"
|
||||
% len(NEW_CONCEPTS))
|
||||
|
||||
# ---- 4. append the objects produced by the splits ------------------------
|
||||
todo_o = [o for o in NEW_OBJECTS
|
||||
if not re.search(r'^%s a ' % re.escape(o[0]), text, re.M)]
|
||||
tail = re.search(r'\n(?=ex:M_\w+ a pr:Metric ;|ex:DO_\w+ a pr:DataObject ;)', text)
|
||||
if tail and todo_o:
|
||||
blocks = "\n".join(object_block(*o) for o in todo_o)
|
||||
text = (text[:tail.start()] + "\n\n" +
|
||||
"# --- Business Objects from the v1.1 splits ---------------------------\n" +
|
||||
"# 'Customer & Outlet' and 'Currency & Exchange Rates' had no single\n" +
|
||||
"# subject. Split proposed to DD-04 and DD-16, hence TO_ARBITRATE.\n" +
|
||||
blocks + "\n" + text[tail.start():])
|
||||
report.append("inserted %d Business Objects from splits" % len(todo_o))
|
||||
elif not todo_o:
|
||||
report.append("Business Objects from splits already present, skipped")
|
||||
|
||||
# ---- report --------------------------------------------------------------
|
||||
print()
|
||||
print("SODH ARBITRATION %s" % ("APPLY" if apply_changes else "DRY RUN"))
|
||||
print("=" * W)
|
||||
for line in report:
|
||||
print(" " + line)
|
||||
print("=" * W)
|
||||
doubled = []
|
||||
for m in re.finditer(r'^(ex:\w+) a pr:\w+ ;(.*?)\.\s*\n', text, re.S | re.M):
|
||||
for prop in ("pr:hasName", "pr:aboutConcept", "pr:belongsTo"):
|
||||
if len(re.findall(r'(?<![\w:])%s\s' % re.escape(prop), m.group(2))) > 1:
|
||||
doubled.append("%s/%s" % (m.group(1), prop))
|
||||
print(" duplicated single-valued properties: %s"
|
||||
% (", ".join(doubled) if doubled else "none"))
|
||||
bo = len(re.findall(r'a pr:BusinessObject', text))
|
||||
bc = len(re.findall(r'a pr:BusinessConcept', text))
|
||||
print(" Business Objects %d | Business Concepts %d | composedOf left %d"
|
||||
% (bo, bc, len(re.findall(r'pr:composedOf', text))))
|
||||
print(" Business Objects still carrying a definition: %d"
|
||||
% len([b for b in re.split(r'\.\s*\n(?=\S)', text)
|
||||
if "pr:BusinessObject" in b and "pr:hasBusinessDefinition" in b]))
|
||||
print("=" * W)
|
||||
|
||||
if text != original and apply_changes:
|
||||
shutil.copy2(TARGET, TARGET + ".bak")
|
||||
open(TARGET, "w", encoding="utf-8").write(text)
|
||||
print(" written, backup at %s.bak" % os.path.basename(TARGET))
|
||||
elif text != original:
|
||||
print(" dry run -- re-run with --apply to write")
|
||||
print()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,482 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
SODH - BR-013 alignment, last step of the v1.1 migration
|
||||
=========================================================
|
||||
Regenerates the Metric and Data Element sections of instances/sodh.ttl from the
|
||||
v0.7 back-doc, which has been ahead of the TTL since the divergence found at
|
||||
audit time. This is the migration that puts the flow back the right way round:
|
||||
after it, the TTL is the source and the workbook becomes a generated artefact.
|
||||
|
||||
USAGE
|
||||
python3 scripts/apply_br013_sodh.py # dry run
|
||||
python3 scripts/apply_br013_sodh.py --apply # rewrite, .bak kept
|
||||
|
||||
WHAT IT FIXES
|
||||
37 BLOCKING DataElementMeaningShape -- no element had a route to meaning
|
||||
15 BLOCKING physicalName on a Metric -- layer leak
|
||||
15 WARNING hasGranularity on a Metric
|
||||
6 WARNING monitoredBy on a Data Object
|
||||
|
||||
WHAT IT DOES
|
||||
1. 15 metrics -> 10 mother metrics carrying the harmonized calculation
|
||||
rule, each with a formula, a unit and a measured Concept. No physical
|
||||
name, no granularity: both belong elsewhere now.
|
||||
2. 95 measure Data Elements generated, each computedBy its mother metric,
|
||||
each carrying the physical name the metric used to hold.
|
||||
3. 37 dimensional Data Elements given a represents towards their Concept.
|
||||
With 2, this closes the XOR: every element reaches meaning by exactly
|
||||
one route.
|
||||
4. Three concepts created because the XOR forces them to be named --
|
||||
Country, Marketing Entity, Fiscal Period had elements but no notion.
|
||||
TO_ARBITRATE like the others.
|
||||
5. Steward removed from the 6 Data Objects: it is inherited now.
|
||||
6. hasGrainElement on the 4 DIMENSION Data Objects only.
|
||||
|
||||
WHY FACT TABLES GET NO GRAIN ELEMENTS
|
||||
A fact table's grain is carried by its foreign keys, and OW-006 says a
|
||||
foreign key is not a Data Element -- so a fact object has no element of its
|
||||
own to point at. Its grain is already expressed, by references towards the
|
||||
dimension objects it joins. Forcing hasGrainElement onto facts would mean
|
||||
either breaking OW-006 or pointing at another object's elements, which
|
||||
GrainConsistencyShape rejects. Nothing is lost: the information is there.
|
||||
"""
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
TTL = os.path.join(REPO, "instances", "sodh.ttl")
|
||||
BACKDOC = os.path.join(REPO, "instances", "SODH_data.xlsx")
|
||||
W = 78
|
||||
|
||||
# ---- mother metric -> the Concept it measures ------------------------------
|
||||
METRIC_CONCEPT = {
|
||||
"M-06.01-001": "ex:BC_06_01_001", # Sell Out Volume -> Sell Out
|
||||
"M-06.01-002": "ex:BC_06_01_001", # Sell Out Value -> Sell Out
|
||||
"M-06.01-003": "ex:BC_06_01_004", # Numeric Distribution -> Retail Distribution
|
||||
"M-06.01-004": "ex:BC_06_01_004", # Weighted Distribution -> Retail Distribution
|
||||
"M-06.01-005": "ex:BC_06_01_004", # Total Distribution Pts -> Retail Distribution
|
||||
"M-06.01-006": "ex:BC_06_01_002", # Forward Stock Volume -> Retailer Stock
|
||||
"M-06.01-007": "ex:BC_06_01_002", # Days of Coverage -> Retailer Stock
|
||||
"M-06.01-008": "ex:BC_06_01_002", # Stock Share -> Retailer Stock
|
||||
"M-06.01-009": "ex:BC_06_01_005", # Baseline Volume -> Sell Out Baseline
|
||||
"M-06.01-010": "ex:BC_06_01_005", # Incremental Volume -> Sell Out Baseline
|
||||
}
|
||||
|
||||
# ---- dimensional element -> the Concept it represents ----------------------
|
||||
# Exact ids first, then id prefixes. Every dimensional element must land
|
||||
# somewhere: that is what the XOR is for.
|
||||
DE_CONCEPT_EXACT = {
|
||||
"DE-04.01-0006": "ex:BC_04_01_002", # Outlet Type -> Outlet
|
||||
"DE-04.01-0007": "ex:BC_04_01_002", # Channel -> Outlet
|
||||
"DE-04.01-0010": "ex:BC_04_01_002", # Point of Sale Count -> Outlet
|
||||
"DE-04.01-0008": "ex:BC_04_03_001", # Country Code -> Country
|
||||
"DE-04.01-0009": "ex:BC_04_03_001", # Country Name -> Country
|
||||
"DE-16.01-0001": "ex:BC_16_01_001", # Currency Code -> Currency
|
||||
"DE-16.01-0002": "ex:BC_16_01_002", # FX Rate to-Euro -> Exchange Rate
|
||||
"DE-16.01-0003": "ex:BC_16_01_002", # FX Rate to-USD -> Exchange Rate
|
||||
"DE-16.02-0001": "ex:BC_16_03_001", # Marketing Entity -> Marketing Entity
|
||||
"DE-16.02-0002": "ex:BC_16_02_001", # Fiscal Period Code -> Fiscal Period
|
||||
"DE-16.02-0003": "ex:BC_16_02_001", # Fiscal Quarter Code -> Fiscal Period
|
||||
}
|
||||
DE_CONCEPT_PREFIX = [
|
||||
("DE-10.01", "ex:BC_10_01_001"), # product hierarchy -> Product
|
||||
("DE-04.01", "ex:BC_04_01_001"), # trade hierarchy -> Customer
|
||||
("DE-21.01", "ex:BC_21_01_001"), # calendar -> Calendar Date
|
||||
("DE-05.01", "ex:BC_05_01_001"), # promotion flags -> Promotion
|
||||
]
|
||||
|
||||
# ---- concepts the XOR forces us to name ------------------------------------
|
||||
NEW_CONCEPTS = [
|
||||
# Final identifiers from the outset. Creating them under one id and
|
||||
# renumbering them later broke idempotence: the second run no longer found
|
||||
# the original id and recreated the concept alongside the renamed one.
|
||||
("ex:BC_04_03_001", "BC-04.03-001", "Country", "ex:DD_04", "ex:ST_DD_04",
|
||||
"A sovereign territory used as the geographic frame for retail measurement, "
|
||||
"identified by its ISO 3166 code. Distinct from Customer and Outlet: it is where "
|
||||
"they operate, not what they are."),
|
||||
("ex:BC_16_03_001", "BC-16.03-001", "Marketing Entity", "ex:DD_16", "ex:ST_DD_16",
|
||||
"An organisational unit of the group holding commercial responsibility for a market, "
|
||||
"and the level at which financial results are consolidated."),
|
||||
("ex:BC_16_02_001", "BC-16.02-001", "Fiscal Period", "ex:DD_16", "ex:ST_DD_16",
|
||||
"A reporting interval of the Pernod Ricard fiscal year, which runs July to June. "
|
||||
"Distinct from Calendar Date: the fiscal frame does not align with the Gregorian one."),
|
||||
]
|
||||
|
||||
# ---- grain of the DIMENSION objects (fact grain lives in references) -------
|
||||
GRAIN = {
|
||||
"ex:DO_10_01_001": ["ex:DE_10_01_0005"], # Product Dimension -> SKU Code
|
||||
"ex:DO_04_01_001": ["ex:DE_04_01_0005"], # Customer Dimension -> Customer Tier-2 Code
|
||||
"ex:DO_16_01_001": ["ex:DE_16_01_0001"], # Currency Dimension -> Currency Code
|
||||
"ex:DO_21_01_001": ["ex:DE_21_01_0001"], # Calendar Dimension -> Calendar Date
|
||||
}
|
||||
|
||||
UNIT_FIX = {"9L / L": "9L", "Currency EUR/USD/LC": "EUR", "%": "%",
|
||||
"Index": "Index", "Days": "Days"}
|
||||
|
||||
|
||||
def iri(ident):
|
||||
"""BO-06.01-001 -> ex:BO_06_01_001"""
|
||||
return "ex:" + ident.replace("-", "_", 1).replace(".", "_").replace("-", "_")
|
||||
|
||||
|
||||
def esc(text):
|
||||
return text.replace("\\", "\\\\").replace('"', '\\"')
|
||||
|
||||
|
||||
def _xlsx_rows(path):
|
||||
"""
|
||||
Minimal .xlsx reader: zipfile + ElementTree, no third-party dependency.
|
||||
|
||||
openpyxl would do this in three lines, but the validation venv is already
|
||||
pinned tightly (pyshacl 0.26.0 for Python 3.9) and adding a dependency to a
|
||||
migration script that runs once is a poor trade. Handles what a back-doc
|
||||
needs: shared strings, inline strings, numbers, and sheet names.
|
||||
"""
|
||||
import zipfile
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
NS = "{http://schemas.openxmlformats.org/spreadsheetml/2006/main}"
|
||||
REL = "{http://schemas.openxmlformats.org/officeDocument/2006/relationships}"
|
||||
PKG = "{http://schemas.openxmlformats.org/package/2006/relationships}"
|
||||
|
||||
with zipfile.ZipFile(path) as z:
|
||||
shared = []
|
||||
if "xl/sharedStrings.xml" in z.namelist():
|
||||
for si in ET.fromstring(z.read("xl/sharedStrings.xml")):
|
||||
shared.append("".join(t.text or "" for t in si.iter(NS + "t")))
|
||||
|
||||
rels = {}
|
||||
for rel in ET.fromstring(z.read("xl/_rels/workbook.xml.rels")):
|
||||
rels[rel.get("Id")] = rel.get("Target").lstrip("/")
|
||||
|
||||
sheets = []
|
||||
for sh in ET.fromstring(z.read("xl/workbook.xml")).iter(NS + "sheet"):
|
||||
target = rels.get(sh.get(REL + "id"), "")
|
||||
if not target.startswith("xl/"):
|
||||
target = "xl/" + target
|
||||
sheets.append((sh.get("name"), target))
|
||||
|
||||
for name, target in sheets:
|
||||
if target not in z.namelist():
|
||||
continue
|
||||
rows = []
|
||||
for row in ET.fromstring(z.read(target)).iter(NS + "row"):
|
||||
cells = []
|
||||
for c in row.iter(NS + "c"):
|
||||
v = c.find(NS + "v")
|
||||
if c.get("t") == "s" and v is not None:
|
||||
cells.append(shared[int(v.text)])
|
||||
elif c.get("t") == "inlineStr":
|
||||
cells.append("".join(t.text or "" for t in c.iter(NS + "t")))
|
||||
else:
|
||||
cells.append(v.text if v is not None else "")
|
||||
rows.append([(x or "").strip() for x in cells])
|
||||
yield name, rows
|
||||
|
||||
|
||||
def _text_rows(path):
|
||||
"""Tab-separated fallback, for back-docs exported as plain text."""
|
||||
sheet, rows = None, []
|
||||
for raw in open(path, encoding="utf-8", errors="replace"):
|
||||
if raw.startswith("## Sheet:"):
|
||||
if sheet:
|
||||
yield sheet, rows
|
||||
sheet, rows = raw.split(":", 1)[1].strip(), []
|
||||
elif sheet is not None:
|
||||
rows.append([c.strip() for c in raw.rstrip("\n").split("\t")])
|
||||
if sheet:
|
||||
yield sheet, rows
|
||||
|
||||
|
||||
def parse_backdoc(path):
|
||||
"""
|
||||
Read the back-doc, whichever form it takes.
|
||||
|
||||
A real .xlsx is a ZIP (magic PK\x03\x04); some pipelines hand over a
|
||||
tab-separated text export of the same content. Sniff rather than assume:
|
||||
guessing from the extension is what made this script fail the first time.
|
||||
"""
|
||||
with open(path, "rb") as fh:
|
||||
is_zip = fh.read(4) == b"PK\x03\x04"
|
||||
reader = _xlsx_rows if is_zip else _text_rows
|
||||
|
||||
out = {"metrics": [], "elements": []}
|
||||
for name, rows in reader(path):
|
||||
key = None
|
||||
if name.strip().startswith("6"):
|
||||
key = "metrics"
|
||||
elif name.strip().startswith("7"):
|
||||
key = "elements"
|
||||
if not key:
|
||||
continue
|
||||
for cells in rows:
|
||||
if len(cells) < 5 or not cells[0]:
|
||||
continue
|
||||
if key == "metrics" and cells[0].startswith("M-"):
|
||||
out[key].append(cells)
|
||||
elif key == "elements" and cells[0].startswith("DE-"):
|
||||
out[key].append(cells)
|
||||
if not out["metrics"] or not out["elements"]:
|
||||
raise SystemExit(
|
||||
"Back-doc read but empty: %d metrics, %d elements.\n"
|
||||
"Expected sheets starting with '6.' (Mother Metrics) and '7.' "
|
||||
"(Data Elements) in %s" % (len(out["metrics"]), len(out["elements"]), path))
|
||||
return out
|
||||
|
||||
|
||||
def concept_for(de_id):
|
||||
if de_id in DE_CONCEPT_EXACT:
|
||||
return DE_CONCEPT_EXACT[de_id]
|
||||
for prefix, bc in DE_CONCEPT_PREFIX:
|
||||
if de_id.startswith(prefix):
|
||||
return bc
|
||||
return None
|
||||
|
||||
|
||||
def build_metrics(metrics):
|
||||
out = ["# --- mother metrics, BR-013 -----------------------------------------",
|
||||
"# One harmonized calculation rule each. The granular variants are Data",
|
||||
"# Elements linked by computedBy. No physical name (layer leak) and no",
|
||||
"# granularity (a calculation rule has no rows).", ""]
|
||||
for r in metrics:
|
||||
mid, name, bo, formula, unit = r[0], r[1], r[2], r[3], r[4]
|
||||
unit = UNIT_FIX.get(unit.split(" (")[0], unit.split(" (")[0])
|
||||
out.append("%s a pr:Metric ;" % iri(mid))
|
||||
out.append(' pr:hasIdentifier "%s" ; pr:hasName "%s" ;' % (mid, esc(name)))
|
||||
out.append(" pr:owningDomain ex:DD_06 ; pr:ownedBy ex:ST_DD_06 ;")
|
||||
out.append(' pr:hasFormula "%s" ;' % esc(formula))
|
||||
out.append(' pr:hasUnit "%s" ;' % esc(unit))
|
||||
out.append(" pr:measures %s ;" % METRIC_CONCEPT[mid])
|
||||
out.append(' pr:hasStatus "DRAFT" ; pr:hasVersion "1.1" ; '
|
||||
'pr:hasSource "SODH back-doc v0.7" .')
|
||||
return "\n".join(out)
|
||||
|
||||
|
||||
def build_elements(elements):
|
||||
measure, dimensional = [], []
|
||||
for r in elements:
|
||||
(deid, name, domain, computed, fmt, unit, source, phys, do) = r[:9]
|
||||
dom = "ex:DD_" + deid.split("-")[1].split(".")[0]
|
||||
block = ["%s a pr:DataElement ;" % iri(deid),
|
||||
' pr:hasIdentifier "%s" ; pr:hasName "%s" ;' % (deid, esc(name)),
|
||||
" pr:owningDomain %s ;" % dom,
|
||||
' pr:hasFormat "%s" ;' % esc(fmt)]
|
||||
if unit and unit != "-":
|
||||
block.append(' pr:hasUnit "%s" ;' % esc(unit))
|
||||
if phys and phys != "-":
|
||||
block.append(' pr:physicalName "%s" ;' % esc(phys))
|
||||
if computed == "(dimensional)":
|
||||
bc = concept_for(deid)
|
||||
if not bc:
|
||||
raise SystemExit("No concept mapped for %s -- the XOR would fail." % deid)
|
||||
block.append(" pr:represents %s ;" % bc)
|
||||
block.append(' pr:hasStatus "DRAFT" ; pr:hasVersion "1.1" ; '
|
||||
'pr:hasSource "%s" .' % esc(source))
|
||||
dimensional.append("\n".join(block))
|
||||
else:
|
||||
block.append(' pr:hasStatus "DRAFT" ; pr:hasVersion "1.1" ; '
|
||||
'pr:hasSource "%s" .' % esc(source))
|
||||
measure.append("\n".join(block))
|
||||
return measure, dimensional
|
||||
|
||||
|
||||
def build_computed_by(elements):
|
||||
"""Metric -> its granular elements. Asserted from the metric side."""
|
||||
by_metric = {}
|
||||
name_to_id = {r[1]: r[0] for r in parse_backdoc(BACKDOC)["metrics"]}
|
||||
for r in elements:
|
||||
if r[3] == "(dimensional)":
|
||||
continue
|
||||
mid = name_to_id.get(r[3])
|
||||
if mid:
|
||||
by_metric.setdefault(mid, []).append(iri(r[0]))
|
||||
out = ["# --- BR-013 wiring: each mother metric and the variants that implement it",
|
||||
""]
|
||||
for mid, des in by_metric.items():
|
||||
chunks = [des[i:i + 4] for i in range(0, len(des), 4)]
|
||||
lines = [" , ".join(c) for c in chunks]
|
||||
out.append("%s pr:computedBy %s ." % (iri(mid), " ,\n ".join(lines)))
|
||||
return "\n".join(out), sum(len(v) for v in by_metric.values())
|
||||
|
||||
|
||||
def rebuild_has_element(text, elements):
|
||||
"""
|
||||
Re-attach every Data Element to its Data Object, from the back-doc column.
|
||||
|
||||
Without this the 95 generated measure elements would be orphans: no path to
|
||||
the physical layer, and no steward, since stewardship is inherited through
|
||||
the object. Not caught by any shape -- which is why it is worth doing here
|
||||
rather than waiting for the validator to complain.
|
||||
"""
|
||||
by_do = {}
|
||||
for r in elements:
|
||||
do = r[8].strip()
|
||||
if do and do != "-":
|
||||
by_do.setdefault(iri(do), []).append(iri(r[0]))
|
||||
n = 0
|
||||
for do, des in by_do.items():
|
||||
pat = re.compile(r'(^%s a pr:DataObject ;.*?)\n\s*pr:hasElement[^;]*;' % re.escape(do),
|
||||
re.S | re.M)
|
||||
chunks = [des[i:i + 4] for i in range(0, len(des), 4)]
|
||||
clause = "\n pr:hasElement " + " ,\n ".join(" , ".join(c) for c in chunks) + " ;"
|
||||
if pat.search(text):
|
||||
text = pat.sub(lambda m: m.group(1) + clause, text, count=1)
|
||||
n += 1
|
||||
return text, n, {k: len(v) for k, v in by_do.items()}
|
||||
|
||||
|
||||
def rebuild_has_metric(text, metrics):
|
||||
"""
|
||||
Re-point hasMetric on the Business Objects at the mother metrics.
|
||||
|
||||
Removing the old metric definitions is not enough: the Business Objects
|
||||
still name them, and because hasMetric has rdfs:range pr:Metric, RDFS
|
||||
entailment types those dangling IRIs as metrics. They then become focus
|
||||
nodes carrying no identifier, no name, no formula -- ten ghosts, six
|
||||
violations each. Deleting a subject means deleting what points at it.
|
||||
"""
|
||||
by_bo = {}
|
||||
for r in metrics:
|
||||
by_bo.setdefault(iri(r[2]), []).append(iri(r[0]))
|
||||
|
||||
text = re.sub(r'\n\s*pr:hasMetric[^;]*;', '', text)
|
||||
n = 0
|
||||
for bo, ms in by_bo.items():
|
||||
pat = re.compile(r'(^%s a pr:BusinessObject ;.*?)(\n\s*pr:hasStatus)' % re.escape(bo),
|
||||
re.S | re.M)
|
||||
if pat.search(text):
|
||||
text = pat.sub(r'\1\n pr:hasMetric %s ;\2' % " , ".join(ms), text, count=1)
|
||||
n += 1
|
||||
return text, n, by_bo
|
||||
|
||||
|
||||
def concept_block(c):
|
||||
i, ident, name, dom, st, definition = c
|
||||
return ("%s a pr:BusinessConcept ;\n"
|
||||
' pr:hasIdentifier "%s" ; pr:hasName "%s" ;\n'
|
||||
" pr:owningDomain %s ; pr:ownedBy %s ;\n"
|
||||
' pr:hasBusinessDefinition "%s" ;\n'
|
||||
' pr:arbitrationStatus "TO_ARBITRATE" ;\n'
|
||||
' pr:hasStatus "DRAFT" ; pr:hasVersion "1.1" ; '
|
||||
'pr:hasSource "DGO proposal, pending ratification by the owning domain" .'
|
||||
% (i, ident, esc(name), dom, st, esc(definition)))
|
||||
|
||||
|
||||
def main():
|
||||
apply_changes = "--apply" in sys.argv
|
||||
for f in (TTL, BACKDOC):
|
||||
if not os.path.exists(f):
|
||||
print("Not found: %s" % f)
|
||||
sys.exit(2)
|
||||
|
||||
doc = parse_backdoc(BACKDOC)
|
||||
text = original = open(TTL, encoding="utf-8").read()
|
||||
report = []
|
||||
|
||||
# 1. drop the old metric and data element blocks
|
||||
n_old_m = len(re.findall(r'^ex:M_\w+ a pr:Metric ;', text, re.M))
|
||||
n_old_de = len(re.findall(r'^ex:DE_\w+ a pr:DataElement ;', text, re.M))
|
||||
text = re.sub(r'^ex:M_\w+ a pr:Metric ;.*?\.\s*\n(?=^ex:|\Z)', '', text, flags=re.S | re.M)
|
||||
text = re.sub(r'^ex:DE_\w+ a pr:DataElement ;.*?\.\s*\n(?=^ex:|\Z)', '', text, flags=re.S | re.M)
|
||||
report.append("removed %d old metrics and %d old data elements" % (n_old_m, n_old_de))
|
||||
|
||||
# 2. steward is inherited, not declared on a Data Object
|
||||
n_st = 0
|
||||
for m in re.finditer(r'^ex:DO_\w+ a pr:DataObject ;.*?\.\s*\n', text, re.S | re.M):
|
||||
block = m.group(0)
|
||||
new = re.sub(r'\s*pr:monitoredBy\s+ex:\w+\s*;', ' ;', block)
|
||||
new = re.sub(r';\s*;', ' ;', new)
|
||||
if new != block:
|
||||
n_st += 1
|
||||
text = text.replace(block, new, 1)
|
||||
report.append("removed monitoredBy from %d Data Objects (inherited now)" % n_st)
|
||||
|
||||
# 3. grain, dimension objects only
|
||||
n_g = 0
|
||||
for do, grain in GRAIN.items():
|
||||
pat = re.compile(r'(^%s a pr:DataObject ;.*?)(\n\s*pr:hasStatus)' % re.escape(do),
|
||||
re.S | re.M)
|
||||
if pat.search(text):
|
||||
text = pat.sub(r'\1\n pr:hasGrainElement %s ;\2' % " , ".join(grain), text, count=1)
|
||||
n_g += 1
|
||||
report.append("grain declared on %d dimension Data Objects "
|
||||
"(fact grain stays in references)" % n_g)
|
||||
|
||||
# 4. the three concepts the XOR forces us to name
|
||||
# idempotent: a concept already in the file is left alone. Duplicated
|
||||
# blocks would be INVISIBLE to SHACL -- identical triples merge under RDF
|
||||
# set semantics, so the graph validates while the file carries redundant
|
||||
# text. A script that can be re-run must check before it inserts.
|
||||
todo = [c for c in NEW_CONCEPTS
|
||||
if not re.search(r'^%s a ' % re.escape(c[0]), text, re.M)]
|
||||
if len(todo) < len(NEW_CONCEPTS):
|
||||
report.append("skipped %d concept(s) already present"
|
||||
% (len(NEW_CONCEPTS) - len(todo)))
|
||||
anchor = re.search(r'\n(?=ex:BO_\w+ a pr:BusinessObject ;)', text)
|
||||
blocks = "\n".join(concept_block(c) for c in todo)
|
||||
if todo:
|
||||
text = (text[:anchor.start()] + "\n\n"
|
||||
+ "# --- concepts required by the meaning XOR ---------------------------\n"
|
||||
+ "# Country, Marketing Entity and Fiscal Period had Data Elements but no\n"
|
||||
+ "# notion behind them. The rule forced them to be named.\n"
|
||||
+ blocks + "\n" + text[anchor.start():])
|
||||
report.append("added %d concepts required by the XOR (TO_ARBITRATE)" % len(todo))
|
||||
|
||||
# 5. metrics, elements, wiring
|
||||
measure, dimensional = build_elements(doc["elements"])
|
||||
wiring, n_wired = build_computed_by(doc["elements"])
|
||||
text = text.rstrip() + "\n\n\n" + build_metrics(doc["metrics"]) + "\n\n"
|
||||
text += ("# --- dimensional data elements --------------------------------------\n"
|
||||
"# Each represents the Concept it carries: the first of the two routes\n"
|
||||
"# to business meaning.\n\n" + "\n".join(dimensional) + "\n\n")
|
||||
text += ("# --- granular measure data elements ---------------------------------\n"
|
||||
"# Same calculation rule as their mother metric, different analysis\n"
|
||||
"# context. They reach meaning through computedBy, never directly.\n\n"
|
||||
+ "\n".join(measure) + "\n\n" + wiring + "\n")
|
||||
report.append("wrote %d mother metrics, %d dimensional and %d measure elements"
|
||||
% (len(doc["metrics"]), len(dimensional), len(measure)))
|
||||
report.append("wired %d computedBy links" % n_wired)
|
||||
|
||||
text, n_bo, per_bo = rebuild_has_metric(text, doc["metrics"])
|
||||
report.append("re-pointed hasMetric on %d Business Objects (%s)"
|
||||
% (n_bo, ", ".join("%s=%d" % (k.replace("ex:BO_", "BO-"), len(v))
|
||||
for k, v in sorted(per_bo.items()))))
|
||||
|
||||
text, n_do, counts = rebuild_has_element(text, doc["elements"])
|
||||
report.append("re-attached elements to %d Data Objects (%s)"
|
||||
% (n_do, ", ".join("%s=%d" % (k.replace("ex:DO_", "DO-"), v)
|
||||
for k, v in sorted(counts.items()))))
|
||||
|
||||
print()
|
||||
print("SODH BR-013 ALIGNMENT %s" % ("APPLY" if apply_changes else "DRY RUN"))
|
||||
print("=" * W)
|
||||
for line in report:
|
||||
print(" " + line)
|
||||
print("=" * W)
|
||||
defined = set(re.findall(r'^(ex:M_\w+) a pr:Metric ;', text, re.M))
|
||||
referenced = set(re.findall(r'ex:M_\w+', text))
|
||||
dangling = referenced - defined
|
||||
print(" dangling metric references: %s"
|
||||
% (", ".join(sorted(dangling)) if dangling else "none"))
|
||||
print(" Metrics %d | Data Elements %d | Concepts %d | computedBy %d | "
|
||||
"physicalName on Metric %d"
|
||||
% (len(re.findall(r'a pr:Metric', text)),
|
||||
len(re.findall(r'a pr:DataElement', text)),
|
||||
len(re.findall(r'a pr:BusinessConcept', text)),
|
||||
n_wired,
|
||||
len(re.findall(r'a pr:Metric ;[^.]*?physicalName', text, re.S))))
|
||||
print("=" * W)
|
||||
|
||||
if text != original and apply_changes:
|
||||
shutil.copy2(TTL, TTL + ".bak")
|
||||
open(TTL, "w", encoding="utf-8").write(text)
|
||||
print(" written, backup at %s.bak" % os.path.basename(TTL))
|
||||
elif text != original:
|
||||
print(" dry run -- re-run with --apply to write")
|
||||
print()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,202 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
SODH - data fixes v1.2
|
||||
=======================
|
||||
Four corrections found while reviewing the viewer.
|
||||
|
||||
USAGE
|
||||
python3 scripts/apply_fixes_v1_2.py # dry run
|
||||
python3 scripts/apply_fixes_v1_2.py --apply # rewrite, .bak kept
|
||||
|
||||
1. DATA DOMAIN OWNERS
|
||||
Six named owners added, plus the hasDomainOwner link. Ownership was
|
||||
readable only through the DGL until now, which conflates two roles: the
|
||||
owner is accountable for the domain, the DGL runs its governance.
|
||||
|
||||
2. hasSource CARRIED TWO MEANINGS
|
||||
On a Data Element it named the system of origin (MDM Product, Group
|
||||
Finance). On a Data Object it named the physical location (SODH Gold /
|
||||
F_SELL_OUT). Same property, two concepts -- the defect we removed from
|
||||
composedOf, still present one layer down. Split: physicalName holds the
|
||||
relation name, hasSource holds the system. This is also what makes the
|
||||
physical layer derivable: F_SELL_OUT plus 132 column names.
|
||||
|
||||
3. METRIC UNITS ARE PLURAL
|
||||
Five mother metrics compute elements in several units (9L and L, EUR and
|
||||
LC and USD). A metric states every unit its variants are expressed in; a
|
||||
Data Element states exactly one. Recomputed from the elements rather than
|
||||
asserted, so the two can never drift.
|
||||
|
||||
4. Sell Out Distribution -> Retail Distribution, matching the concept: sell
|
||||
out counts units sold, distribution measures presence in store.
|
||||
"""
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
TTL = os.path.join(REPO, "instances", "sodh.ttl")
|
||||
W = 78
|
||||
|
||||
OWNERS = [
|
||||
("ex:DDO_DD_06", "Ludovic Cottier", "ex:DD_06"),
|
||||
("ex:DDO_DD_04", "David Carpio", "ex:DD_04"),
|
||||
("ex:DDO_DD_10", "Alberto Lupano", "ex:DD_10"),
|
||||
("ex:DDO_DD_05", "David Carpio", "ex:DD_05"),
|
||||
("ex:DDO_DD_16", "Julien Soisson", "ex:DD_16"),
|
||||
("ex:DDO_DD_21", "Vincent Meunier", "ex:DD_21"),
|
||||
]
|
||||
|
||||
RENAME = {"ex:BO_06_01_003": ("Sell Out Distribution", "Retail Distribution")}
|
||||
|
||||
# system of origin, once the physical location is taken out of it
|
||||
SOURCE_FIX = {
|
||||
"Snowflake (SODH Gold)": "SODH Gold",
|
||||
"SODH Gold (flag)": "SODH Gold",
|
||||
}
|
||||
|
||||
|
||||
def clauses(body):
|
||||
out, buf, i, n = [], [], 0, len(body)
|
||||
while i < n:
|
||||
c = body[i]
|
||||
if c == '"':
|
||||
j = i + 1
|
||||
while j < n and body[j] != '"':
|
||||
j += 2 if body[j] == "\\" else 1
|
||||
j = min(j + 1, n)
|
||||
buf.append(body[i:j]); i = j; continue
|
||||
if c == ";":
|
||||
out.append("".join(buf)); buf = []; i += 1; continue
|
||||
buf.append(c); i += 1
|
||||
out.append("".join(buf))
|
||||
return out
|
||||
|
||||
|
||||
def set_prop(text, subject, prop, value):
|
||||
"""Clause-level set: survives mid-line properties and semicolons in literals."""
|
||||
pat = re.compile(r'(^%s a pr:\w+ ;)(.*?)(\.\s*\n)' % re.escape(subject), re.S | re.M)
|
||||
m = pat.search(text)
|
||||
if not m:
|
||||
return text, False
|
||||
kept = [c for c in clauses(m.group(2))
|
||||
if c.strip() and not re.match(r'\s*%s\s' % re.escape(prop), c)]
|
||||
if value is not None:
|
||||
kept.insert(0, "\n %s %s " % (prop, value))
|
||||
body = " ;".join(kept).rstrip() + "\n "
|
||||
return text[:m.start()] + m.group(1) + body + m.group(3) + text[m.end():], True
|
||||
|
||||
|
||||
def get_prop(text, subject, prop):
|
||||
m = re.search(r'^%s a pr:\w+ ;(.*?)\.\s*\n' % re.escape(subject), text, re.S | re.M)
|
||||
if not m:
|
||||
return None
|
||||
for c in clauses(m.group(1)):
|
||||
mm = re.match(r'\s*%s\s+(.*)$' % re.escape(prop), c, re.S)
|
||||
if mm:
|
||||
return mm.group(1).strip()
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
apply_changes = "--apply" in sys.argv
|
||||
if not os.path.exists(TTL):
|
||||
print("Not found: %s" % TTL); sys.exit(2)
|
||||
text = original = open(TTL, encoding="utf-8").read()
|
||||
report = []
|
||||
|
||||
# ---- 1. domain owners -------------------------------------------------
|
||||
todo = [o for o in OWNERS if not re.search(r'^%s a ' % re.escape(o[0]), text, re.M)]
|
||||
if todo:
|
||||
blocks = "\n".join('%s a pr:DataDomainOwner ; pr:hasName "%s" .' % (i, n)
|
||||
for i, n, _ in todo)
|
||||
anchor = re.search(r'^ex:ST_\w+ a pr:DataSteward', text, re.M)
|
||||
pos = anchor.start() if anchor else 0
|
||||
text = (text[:pos]
|
||||
+ "# --- data domain owners ----------------------------------------------\n"
|
||||
+ "# Accountable for the domain. Distinct from the DGL, who runs its\n"
|
||||
+ "# governance: two roles that hasDGL alone was conflating.\n"
|
||||
+ blocks + "\n\n" + text[pos:])
|
||||
for i, n, dd in OWNERS:
|
||||
text, _ = set_prop(text, dd, "pr:hasDomainOwner", i)
|
||||
report.append("%d data domain owners named and linked" % len(OWNERS))
|
||||
|
||||
# ---- 2. physical location out of hasSource ----------------------------
|
||||
n_do = 0
|
||||
for m in list(re.finditer(r'^(ex:DO_\w+) a pr:DataObject ;', text, re.M)):
|
||||
subj = m.group(1)
|
||||
src = get_prop(text, subj, "pr:hasSource")
|
||||
if src and "/" in src:
|
||||
system, relation = [p.strip().strip('"') for p in src.strip('"').split("/", 1)]
|
||||
text, _ = set_prop(text, subj, "pr:hasSource", '"%s"' % system)
|
||||
text, _ = set_prop(text, subj, "pr:physicalName", '"%s"' % relation)
|
||||
n_do += 1
|
||||
report.append("%d data objects: physical relation moved out of hasSource" % n_do)
|
||||
|
||||
n_de = 0
|
||||
for old, new in SOURCE_FIX.items():
|
||||
text, k = re.subn(r'pr:hasSource "%s"' % re.escape(old),
|
||||
'pr:hasSource "%s"' % new, text)
|
||||
n_de += k
|
||||
report.append("%d source labels normalised to the system of origin" % n_de)
|
||||
|
||||
# ---- 3. metric units, recomputed from the elements ---------------------
|
||||
de_unit = {}
|
||||
for m in re.finditer(r'^(ex:DE_\w+) a pr:DataElement ;(.*?)\.\s*\n', text, re.S | re.M):
|
||||
u = re.search(r'pr:hasUnit\s+"([^"]*)"', m.group(2))
|
||||
if u:
|
||||
de_unit[m.group(1)] = u.group(1)
|
||||
wired = {}
|
||||
for m in re.finditer(r'(ex:M_\w+)\s+pr:computedBy(.*?)\.', text, re.S):
|
||||
wired[m.group(1)] = re.findall(r'ex:DE_\w+', m.group(2))
|
||||
n_m = 0
|
||||
for metric, des in wired.items():
|
||||
units = sorted({de_unit[d] for d in des if d in de_unit and de_unit[d] != "-"})
|
||||
if not units:
|
||||
continue
|
||||
value = " , ".join('"%s"' % u for u in units)
|
||||
before = get_prop(text, metric, "pr:hasUnit")
|
||||
text, ok = set_prop(text, metric, "pr:hasUnit", value)
|
||||
if ok and before != value:
|
||||
n_m += 1
|
||||
report.append("%d metrics now list every unit their elements use" % n_m)
|
||||
|
||||
# ---- 4. rename --------------------------------------------------------
|
||||
for subj, (old, new) in RENAME.items():
|
||||
if get_prop(text, subj, "pr:hasName") == '"%s"' % old:
|
||||
text, _ = set_prop(text, subj, "pr:hasName", '"%s"' % new)
|
||||
report.append("renamed %s -> %s" % (old, new))
|
||||
|
||||
# ---- report -----------------------------------------------------------
|
||||
print()
|
||||
print("SODH DATA FIXES v1.2 %s" % ("APPLY" if apply_changes else "DRY RUN"))
|
||||
print("=" * W)
|
||||
for line in report:
|
||||
print(" " + line)
|
||||
print("=" * W)
|
||||
multi = len(re.findall(r'pr:hasUnit\s+"[^"]*"\s*,', text))
|
||||
phys = len(re.findall(r'^ex:DO_\w+ a pr:DataObject ;(?:(?!\.\s*\n).)*?physicalName',
|
||||
text, re.S | re.M))
|
||||
doubled = []
|
||||
for m in re.finditer(r'^(ex:\w+) a pr:\w+ ;(.*?)\.\s*\n', text, re.S | re.M):
|
||||
for p in ("pr:hasName", "pr:hasSource", "pr:physicalName", "pr:hasDomainOwner"):
|
||||
if len(re.findall(r'(?<![\w:])%s\s' % re.escape(p), m.group(2))) > 1:
|
||||
doubled.append("%s/%s" % (m.group(1), p))
|
||||
print(" metrics with several units %d | data objects with a physical name %d"
|
||||
% (multi, phys))
|
||||
print(" duplicated single-valued properties: %s"
|
||||
% (", ".join(doubled) if doubled else "none"))
|
||||
print("=" * W)
|
||||
|
||||
if text != original and apply_changes:
|
||||
shutil.copy2(TTL, TTL + ".bak")
|
||||
open(TTL, "w", encoding="utf-8").write(text)
|
||||
print(" written, backup at %s.bak" % os.path.basename(TTL))
|
||||
elif text != original:
|
||||
print(" dry run -- re-run with --apply to write")
|
||||
print()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,261 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
SODH - structure proposed for the borrowed domains
|
||||
===================================================
|
||||
Gives the nine TO_ARBITRATE concepts a home: a Business Object and a
|
||||
Sub-Domain each, per the structure arbitrated with Bastien on 2026-07-27.
|
||||
|
||||
USAGE
|
||||
python3 scripts/apply_structure_sodh.py # dry run
|
||||
python3 scripts/apply_structure_sodh.py --apply # rewrite, .bak kept
|
||||
|
||||
STILL A PROPOSAL
|
||||
Everything created here stays TO_ARBITRATE. DD-04, DD-10, DD-16 and DD-21
|
||||
own these perimeters; the DGO is proposing a coherent structure, not
|
||||
ratifying one. Replacing padding with an equally unilateral structure
|
||||
would trade one problem for a tidier version of the same problem. The flag
|
||||
comes off when Helene Puchot, Anas El Kesri and Gaelle Seret say so, and
|
||||
OW-007 blocks publication until then.
|
||||
|
||||
IDENTIFIERS ARE FINAL FROM CREATION
|
||||
Identifiers encode the sub-domain (BO-04.02-001 lives in SD-04.02), so the
|
||||
upstream scripts now mint them correctly rather than having this one
|
||||
renumber afterwards. Renaming after creation broke idempotence: the script
|
||||
that created an object stopped recognising its own output and made a second
|
||||
copy on the next run.
|
||||
|
||||
COMPOUND SUB-DOMAINS SPLIT TOO
|
||||
'Customer & Outlet' and 'Marketing Entity & Fiscal Calendar' have the same
|
||||
defect as the Business Objects that were split earlier: a name joined by
|
||||
'&' has no single subject. Same rule, one level up.
|
||||
"""
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
TTL = os.path.join(REPO, "instances", "sodh.ttl")
|
||||
W = 78
|
||||
|
||||
# No IRI renaming here. Objects are created with their final identifiers by
|
||||
# the upstream scripts: renumbering after the fact broke idempotence, because
|
||||
# the creating script then no longer recognised its own output.
|
||||
# ---- sub-domains ----------------------------------------------------------
|
||||
SD_RENAME = {
|
||||
"ex:SD_04_01": "Customer Structure",
|
||||
"ex:SD_16_01": "Currency & Exchange Rate",
|
||||
"ex:SD_16_02": "Fiscal Calendar",
|
||||
"ex:SD_21_01": "Standard Calendar",
|
||||
}
|
||||
SD_NEW = [
|
||||
("ex:SD_04_02", "SD-04.02", "Outlet & Points of Sale", "ex:DD_04"),
|
||||
("ex:SD_04_03", "SD-04.03", "Geography", "ex:DD_04"),
|
||||
("ex:SD_16_03", "SD-16.03", "Organisation Structure", "ex:DD_16"),
|
||||
]
|
||||
|
||||
# ---- business objects -----------------------------------------------------
|
||||
BO_RENAME = {
|
||||
"ex:BO_10_01_001": "Product Hierarchy",
|
||||
"ex:BO_04_01_001": "Customer Hierarchy",
|
||||
"ex:BO_21_01_001": "Analysis Period",
|
||||
}
|
||||
BO_NEW = [
|
||||
("ex:BO_04_03_001", "BO-04.03-001", "Country", "ex:DD_04", "ex:ST_DD_04",
|
||||
"ex:SD_04_03", "ex:BC_04_03_001"),
|
||||
("ex:BO_16_02_001", "BO-16.02-001", "Fiscal Period", "ex:DD_16", "ex:ST_DD_16",
|
||||
"ex:SD_16_02", "ex:BC_16_02_001"),
|
||||
("ex:BO_16_03_001", "BO-16.03-001", "Marketing Entity", "ex:DD_16", "ex:ST_DD_16",
|
||||
"ex:SD_16_03", "ex:BC_16_03_001"),
|
||||
]
|
||||
|
||||
# ---- where each object finally sits ---------------------------------------
|
||||
BELONGS = {
|
||||
"ex:BO_10_01_001": "ex:SD_10_01",
|
||||
"ex:BO_04_01_001": "ex:SD_04_01",
|
||||
"ex:BO_04_02_001": "ex:SD_04_02",
|
||||
"ex:BO_04_03_001": "ex:SD_04_03",
|
||||
"ex:BO_16_01_001": "ex:SD_16_01",
|
||||
"ex:BO_16_01_002": "ex:SD_16_01",
|
||||
"ex:BO_16_02_001": "ex:SD_16_02",
|
||||
"ex:BO_16_03_001": "ex:SD_16_03",
|
||||
"ex:BO_21_01_001": "ex:SD_21_01",
|
||||
}
|
||||
|
||||
# ---- subject of each borrowed object --------------------------------------
|
||||
ABOUT = {
|
||||
"ex:BO_10_01_001": "ex:BC_10_01_001", # Product Hierarchy -> Product
|
||||
"ex:BO_04_01_001": "ex:BC_04_01_001", # Customer Hierarchy -> Customer
|
||||
"ex:BO_04_02_001": "ex:BC_04_01_002", # Outlet -> Outlet
|
||||
"ex:BO_04_03_001": "ex:BC_04_03_001", # Country -> Country
|
||||
"ex:BO_16_01_001": "ex:BC_16_01_001", # Currency -> Currency
|
||||
"ex:BO_16_01_002": "ex:BC_16_01_002", # Exchange Rate -> Exchange Rate
|
||||
"ex:BO_16_02_001": "ex:BC_16_02_001", # Fiscal Period -> Fiscal Period
|
||||
"ex:BO_16_03_001": "ex:BC_16_03_001", # Marketing Entity -> Marketing Entity
|
||||
"ex:BO_21_01_001": "ex:BC_21_01_001", # Analysis Period -> Calendar Date
|
||||
}
|
||||
|
||||
MARKETING_NOTE = ("Named Marketing Entity, not Management Entity: the element is "
|
||||
"Marketing Entity Code and the column is MARKETING_ENTITY_CD. "
|
||||
"To be confirmed by DD-16.")
|
||||
|
||||
|
||||
def sd_block(i, ident, name, domain):
|
||||
return ('%s a pr:SubDomain ;\n'
|
||||
' pr:hasIdentifier "%s" ; pr:hasName "%s" ;\n'
|
||||
' pr:belongsTo %s ; pr:owningDomain %s ;\n'
|
||||
' pr:arbitrationStatus "TO_ARBITRATE" ;\n'
|
||||
' pr:hasStatus "DRAFT" ; pr:hasVersion "1.1" ; '
|
||||
'pr:hasSource "DGO proposal, split of a compound sub-domain" .'
|
||||
% (i, ident, name, domain, domain))
|
||||
|
||||
|
||||
def bo_block(i, ident, name, domain, steward, sd, bc):
|
||||
return ('%s a pr:BusinessObject ;\n'
|
||||
' pr:hasIdentifier "%s" ; pr:hasName "%s" ;\n'
|
||||
' pr:owningDomain %s ; pr:ownedBy %s ;\n'
|
||||
' pr:belongsTo %s ; pr:monitoredBy %s ;\n'
|
||||
' pr:aboutConcept %s ;\n'
|
||||
' pr:arbitrationStatus "TO_ARBITRATE" ;\n'
|
||||
' pr:hasStatus "DRAFT" ; pr:hasVersion "1.1" ; '
|
||||
'pr:hasSource "DGO proposal, pending ratification by the owning domain" .'
|
||||
% (i, ident, name, domain, steward, sd, steward, bc))
|
||||
|
||||
|
||||
def _split_clauses(body):
|
||||
"""Split a Turtle predicate list on ';' that sit outside string literals."""
|
||||
out, buf, i, n = [], [], 0, len(body)
|
||||
while i < n:
|
||||
c = body[i]
|
||||
if c == '"':
|
||||
j = i + 1
|
||||
while j < n and body[j] != '"':
|
||||
j += 2 if body[j] == "\\" else 1
|
||||
j = min(j + 1, n)
|
||||
buf.append(body[i:j]); i = j; continue
|
||||
if c == ";":
|
||||
out.append("".join(buf)); buf = []; i += 1; continue
|
||||
buf.append(c); i += 1
|
||||
out.append("".join(buf))
|
||||
return out
|
||||
|
||||
|
||||
def set_prop(text, subject, prop, value):
|
||||
"""
|
||||
Set or replace a single-valued property inside a subject block.
|
||||
|
||||
Clause-level, not regex-level: a property can sit mid-line
|
||||
(pr:hasIdentifier "X" ; pr:hasName "Y" ;) and its value can contain a
|
||||
semicolon inside the literal. A line-anchored pattern silently misses both
|
||||
and APPENDS instead of replacing, which produces a second value and a
|
||||
cardinality violation -- exactly what happened on the first run here.
|
||||
"""
|
||||
pat = re.compile(r'(^%s a pr:\w+ ;)(.*?)(\.\s*\n)' % re.escape(subject), re.S | re.M)
|
||||
m = pat.search(text)
|
||||
if not m:
|
||||
return text, False
|
||||
kept = [c for c in _split_clauses(m.group(2))
|
||||
if c.strip() and not re.match(r'\s*%s\s' % re.escape(prop), c)]
|
||||
kept.insert(0, "\n %s %s " % (prop, value))
|
||||
body = " ;".join(kept)
|
||||
if not body.endswith("\n"):
|
||||
body = body.rstrip() + "\n "
|
||||
return text[:m.start()] + m.group(1) + body + m.group(3) + text[m.end():], True
|
||||
|
||||
|
||||
def main():
|
||||
apply_changes = "--apply" in sys.argv
|
||||
if not os.path.exists(TTL):
|
||||
print("Not found: %s" % TTL)
|
||||
sys.exit(2)
|
||||
text = original = open(TTL, encoding="utf-8").read()
|
||||
report = []
|
||||
|
||||
# 2. sub-domains
|
||||
for sd, name in SD_RENAME.items():
|
||||
text, ok = set_prop(text, sd, "pr:hasName", '"%s"' % name)
|
||||
if ok:
|
||||
report.append("renamed %s -> %s" % (sd.replace("ex:SD_", "SD-"), name))
|
||||
|
||||
todo_sd = [s for s in SD_NEW if not re.search(r'^%s a ' % re.escape(s[0]), text, re.M)]
|
||||
if todo_sd:
|
||||
anchor = re.search(r'\n(?=ex:BC_\w+ a pr:BusinessConcept ;|ex:BO_\w+ a pr:BusinessObject ;)',
|
||||
text)
|
||||
blocks = "\n".join(sd_block(*s) for s in todo_sd)
|
||||
text = (text[:anchor.start()] + "\n\n"
|
||||
+ "# --- sub-domains from the v1.1 structure proposal --------------------\n"
|
||||
+ "# 'Customer & Outlet' and 'Marketing Entity & Fiscal Calendar' had the\n"
|
||||
+ "# same defect as the compound Business Objects: a name joined by '&'\n"
|
||||
+ "# has no single subject. Same rule, one level up.\n"
|
||||
+ blocks + "\n" + text[anchor.start():])
|
||||
report.append("created %d sub-domains%s"
|
||||
% (len(todo_sd), "" if todo_sd else " (already present)"))
|
||||
|
||||
# 3. new business objects
|
||||
todo_bo = [b for b in BO_NEW if not re.search(r'^%s a ' % re.escape(b[0]), text, re.M)]
|
||||
if todo_bo:
|
||||
anchor = re.search(r'\n(?=ex:DO_\w+ a pr:DataObject ;|ex:M_\w+ a pr:Metric ;)', text)
|
||||
blocks = "\n".join(bo_block(*b) for b in todo_bo)
|
||||
text = (text[:anchor.start()] + "\n\n"
|
||||
+ "# --- business objects for the borrowed concepts -----------------------\n"
|
||||
+ "# Country, Fiscal Period and Marketing Entity had a concept but nothing\n"
|
||||
+ "# to hold it. Proposed to DD-04 and DD-16, hence TO_ARBITRATE.\n"
|
||||
+ blocks + "\n" + text[anchor.start():])
|
||||
report.append("created %d business objects%s"
|
||||
% (len(todo_bo), "" if todo_bo else " (already present)"))
|
||||
|
||||
# 4. renames, placement, subject
|
||||
for bo, name in BO_RENAME.items():
|
||||
text, _ = set_prop(text, bo, "pr:hasName", '"%s"' % name)
|
||||
report.append("renamed %d business objects to the agreed nomenclature" % len(BO_RENAME))
|
||||
|
||||
n_b = n_a = 0
|
||||
for bo, sd in BELONGS.items():
|
||||
text, ok = set_prop(text, bo, "pr:belongsTo", sd)
|
||||
n_b += ok
|
||||
for bo, bc in ABOUT.items():
|
||||
text, ok = set_prop(text, bo, "pr:aboutConcept", bc)
|
||||
n_a += ok
|
||||
report.append("placed %d objects in their sub-domain, %d subjects confirmed" % (n_b, n_a))
|
||||
|
||||
# 5. the naming caveat, recorded on the object rather than in a side note
|
||||
text, ok = set_prop(text, "ex:BC_16_03_001", "pr:hasBusinessRule", '"%s"' % MARKETING_NOTE)
|
||||
if ok:
|
||||
report.append("recorded the Marketing/Management naming caveat on the concept")
|
||||
|
||||
# ---- report
|
||||
print()
|
||||
print("SODH STRUCTURE PROPOSAL %s" % ("APPLY" if apply_changes else "DRY RUN"))
|
||||
print("=" * W)
|
||||
for line in report:
|
||||
print(" " + line)
|
||||
print("=" * W)
|
||||
defined = set(re.findall(r'^(ex:\w+) a ', text, re.M))
|
||||
dangling = sorted(set(re.findall(r'\bex:\w+', text)) - defined)
|
||||
print(" dangling references: %s" % (", ".join(dangling) if dangling else "none"))
|
||||
for cls in ("SubDomain", "BusinessObject", "BusinessConcept"):
|
||||
ids = re.findall(r'^(ex:\w+) a pr:%s\b' % cls, text, re.M)
|
||||
dup = [i for i in set(ids) if ids.count(i) > 1]
|
||||
print(" %-16s %3d duplicates %d" % (cls, len(ids), len(dup)))
|
||||
doubled = []
|
||||
for m in re.finditer(r'^(ex:\w+) a pr:\w+ ;(.*?)\.\s*\n', text, re.S | re.M):
|
||||
for prop in ("pr:hasName", "pr:hasIdentifier", "pr:belongsTo", "pr:aboutConcept"):
|
||||
if len(re.findall(r'%s\s' % re.escape(prop), m.group(2))) > 1:
|
||||
doubled.append("%s/%s" % (m.group(1), prop))
|
||||
print(" duplicated single-valued properties: %s"
|
||||
% (", ".join(doubled) if doubled else "none"))
|
||||
arb = len(re.findall(r'pr:arbitrationStatus "TO_ARBITRATE"', text))
|
||||
print(" TO_ARBITRATE %3d (nothing publishes until the domains ratify)" % arb)
|
||||
print("=" * W)
|
||||
|
||||
if text != original and apply_changes:
|
||||
shutil.copy2(TTL, TTL + ".bak")
|
||||
open(TTL, "w", encoding="utf-8").write(text)
|
||||
print(" written, backup at %s.bak" % os.path.basename(TTL))
|
||||
elif text != original:
|
||||
print(" dry run -- re-run with --apply to write")
|
||||
print()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
File diff suppressed because it is too large
Load Diff
+997
-749
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,214 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
PR Data Meta Model - instance migration v1.0 -> v1.1
|
||||
=====================================================
|
||||
Rewrites instance files for the T-Box v1.1 vocabulary. Text-level rewriting on
|
||||
purpose: it preserves comments, ordering and formatting, which a parse-and-
|
||||
reserialise round trip through rdflib would destroy -- and those comments carry
|
||||
the arbitration notes.
|
||||
|
||||
USAGE
|
||||
python3 scripts/migrate_instances_v1_1.py # dry run, report only
|
||||
python3 scripts/migrate_instances_v1_1.py --apply # rewrite in place, .bak kept
|
||||
python3 scripts/migrate_instances_v1_1.py --apply f.ttl # a single file
|
||||
|
||||
WHAT IT DOES AUTOMATICALLY
|
||||
namespace /metamodel/0.9/ -> /metamodel/ (unversioned terms)
|
||||
belongsToDomain, belongsToSubDomain -> belongsTo
|
||||
implements -> represents
|
||||
hasSteward -> monitoredBy
|
||||
usesDataObject -> packages
|
||||
DataAsset -> DataInterface
|
||||
hasGrainDimension, hasGranularity -> flagged, not converted
|
||||
|
||||
WHAT IT REFUSES TO DO
|
||||
composedOf -> aboutConcept / usesConcept is NOT automated. Splitting it
|
||||
requires deciding which concept is the SUBJECT of each Business Object, and
|
||||
guessing that is exactly how v0.6 ended up with Calendar declared as being
|
||||
composed of Sell Out. The script lists the choices to be made and leaves
|
||||
composedOf in place until a human answers.
|
||||
|
||||
hasBusinessDefinition on a Business Object is likewise reported, not moved:
|
||||
the text must go to the Concept named by aboutConcept, which does not exist
|
||||
yet at the time this runs.
|
||||
"""
|
||||
import glob
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
INSTANCES = os.path.join(REPO, "instances")
|
||||
W = 78
|
||||
|
||||
# (pattern, replacement, label) -- applied in order, word-bounded
|
||||
RENAMES = [
|
||||
(r"https://ontology\.pernod-ricard\.com/metamodel/0\.9/",
|
||||
"https://ontology.pernod-ricard.com/metamodel/", "namespace 0.9 -> unversioned"),
|
||||
(r"\bpr:belongsToDomain\b", "pr:belongsTo", "belongsToDomain -> belongsTo"),
|
||||
(r"\bpr:belongsToSubDomain\b", "pr:belongsTo", "belongsToSubDomain -> belongsTo"),
|
||||
(r"\bpr:implements\b", "pr:represents", "implements -> represents"),
|
||||
(r"\bpr:hasSteward\b", "pr:monitoredBy", "hasSteward -> monitoredBy"),
|
||||
(r"\bpr:usesDataObject\b", "pr:packages", "usesDataObject -> packages"),
|
||||
(r"\bpr:DataAsset\b", "pr:DataInterface", "DataAsset -> DataInterface"),
|
||||
]
|
||||
|
||||
# things a human must decide, reported per file
|
||||
MANUAL = [
|
||||
(r"\bpr:composedOf\b", "composedOf",
|
||||
"Split into aboutConcept (exactly one -- the SUBJECT) and usesConcept "
|
||||
"(the rest). Not automated: guessing the subject is how v0.6 ended up "
|
||||
"with Calendar composed of Sell Out."),
|
||||
(r"\bpr:isComponentOf\b", "isComponentOf",
|
||||
"Inverse of the above. Re-express as usesConcept from the Business Object."),
|
||||
(r"\bpr:hasGranularity\b", "hasGranularity",
|
||||
"Grain moved off the Metric. Re-express as hasGrainElement on the Data "
|
||||
"Object, pointing at the Data Elements that make a row unique."),
|
||||
(r"\bpr:hasGrainDimension\b", "hasGrainDimension",
|
||||
"Same: hasGrainElement on the Data Object, targeting Data Elements."),
|
||||
]
|
||||
|
||||
|
||||
def subject_blocks(text):
|
||||
"""Yield (subject, block) for each top-level statement."""
|
||||
for block in re.split(r"\.\s*\n(?=\S)", text):
|
||||
m = re.match(r"\s*(\S+)", block)
|
||||
if m:
|
||||
yield m.group(1), block
|
||||
|
||||
|
||||
def business_objects_with_definition(text):
|
||||
"""Business Objects still carrying hasBusinessDefinition -- must move to the Concept."""
|
||||
out = []
|
||||
for subj, block in subject_blocks(text):
|
||||
if "pr:BusinessObject" in block and "pr:hasBusinessDefinition" in block:
|
||||
out.append(subj)
|
||||
return out
|
||||
|
||||
|
||||
def composed_of_map(text):
|
||||
"""Business Object -> list of concepts, so the subject can be chosen."""
|
||||
out = {}
|
||||
for subj, block in subject_blocks(text):
|
||||
if "pr:composedOf" not in block:
|
||||
continue
|
||||
m = re.search(r"pr:composedOf\s+([^;.]+)", block)
|
||||
if m:
|
||||
concepts = [c.strip() for c in m.group(1).split(",") if c.strip()]
|
||||
out[subj] = concepts
|
||||
return out
|
||||
|
||||
|
||||
def process(path, apply_changes):
|
||||
original = open(path, encoding="utf-8").read()
|
||||
text = original
|
||||
applied = []
|
||||
|
||||
for pattern, repl, label in RENAMES:
|
||||
text, n = re.subn(pattern, repl, text)
|
||||
if n:
|
||||
applied.append((label, n))
|
||||
|
||||
pending = []
|
||||
for pattern, label, advice in MANUAL:
|
||||
n = len(re.findall(pattern, original))
|
||||
if n:
|
||||
pending.append((label, n, advice))
|
||||
|
||||
bo_defs = business_objects_with_definition(original)
|
||||
comp = composed_of_map(original)
|
||||
|
||||
print("=" * W)
|
||||
print(os.path.relpath(path, REPO))
|
||||
print("=" * W)
|
||||
|
||||
if applied:
|
||||
print(" AUTOMATIC")
|
||||
for label, n in applied:
|
||||
print(" %-42s %4d occurrence(s)" % (label, n))
|
||||
else:
|
||||
print(" AUTOMATIC nothing to rewrite")
|
||||
|
||||
if pending:
|
||||
print(" MANUAL")
|
||||
for label, n, advice in pending:
|
||||
print(" %-42s %4d occurrence(s)" % (label, n))
|
||||
for line in _wrap(advice, 68):
|
||||
print(" " + line)
|
||||
|
||||
if bo_defs:
|
||||
print(" BUSINESS DEFINITIONS TO MOVE")
|
||||
print(" %d Business Object(s) still carry hasBusinessDefinition." % len(bo_defs))
|
||||
print(" Meaning belongs to the Concept. Move each text to the Concept")
|
||||
print(" named by that object's aboutConcept, then delete it here.")
|
||||
for s in bo_defs[:12]:
|
||||
print(" - %s" % s)
|
||||
if len(bo_defs) > 12:
|
||||
print(" ... and %d more" % (len(bo_defs) - 12))
|
||||
|
||||
if comp:
|
||||
print(" SUBJECT TO CHOOSE (composedOf -> aboutConcept + usesConcept)")
|
||||
for subj, concepts in list(comp.items())[:12]:
|
||||
print(" %s" % subj)
|
||||
print(" candidates: %s" % ", ".join(concepts))
|
||||
if len(comp) > 12:
|
||||
print(" ... and %d more" % (len(comp) - 12))
|
||||
|
||||
changed = text != original
|
||||
if changed and apply_changes:
|
||||
shutil.copy2(path, path + ".bak")
|
||||
open(path, "w", encoding="utf-8").write(text)
|
||||
print(" WRITTEN (backup at %s.bak)" % os.path.basename(path))
|
||||
elif changed:
|
||||
print(" DRY RUN -- re-run with --apply to write")
|
||||
print()
|
||||
return changed, len(pending) + len(bo_defs) + len(comp)
|
||||
|
||||
|
||||
def _wrap(text, width):
|
||||
words, line, out = text.split(), "", []
|
||||
for w in words:
|
||||
if len(line) + len(w) + 1 > width:
|
||||
out.append(line)
|
||||
line = w
|
||||
else:
|
||||
line = (line + " " + w).strip()
|
||||
if line:
|
||||
out.append(line)
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
apply_changes = "--apply" in sys.argv
|
||||
args = [a for a in sys.argv[1:] if not a.startswith("-")]
|
||||
files = ([a if os.path.isabs(a) else os.path.join(REPO, a) for a in args]
|
||||
or sorted(glob.glob(os.path.join(INSTANCES, "*.ttl"))))
|
||||
files = [f for f in files if os.path.exists(f)]
|
||||
|
||||
if not files:
|
||||
print("No instance file found.")
|
||||
sys.exit(2)
|
||||
|
||||
print()
|
||||
print("PR META MODEL - INSTANCE MIGRATION v1.0 -> v1.1 %s"
|
||||
% ("APPLY" if apply_changes else "DRY RUN"))
|
||||
print()
|
||||
|
||||
total_changed = total_manual = 0
|
||||
for f in files:
|
||||
changed, manual = process(f, apply_changes)
|
||||
total_changed += int(changed)
|
||||
total_manual += manual
|
||||
|
||||
print("=" * W)
|
||||
print("%d file(s) rewritten, %d decision(s) left to a human."
|
||||
% (total_changed, total_manual))
|
||||
if total_manual:
|
||||
print("Run the validator after the manual work: the XOR-of-meaning shape")
|
||||
print("on Data Elements will not pass until the concepts are wired.")
|
||||
print("=" * W)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,207 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
SODH - migration v1.1 -> v1.2
|
||||
==============================
|
||||
USAGE
|
||||
python3 scripts/migrate_instances_v1_2.py # dry run
|
||||
python3 scripts/migrate_instances_v1_2.py --apply # rewrite, .bak kept
|
||||
|
||||
WHAT IT DOES
|
||||
1. physicalizedIn -> storedIn everywhere (no instances yet, done for
|
||||
completeness so the term never appears again).
|
||||
2. hasSource removed from every Business Concept. A concept has no source
|
||||
of truth: its authority is its owning domain, its definition belongs in
|
||||
the glossary. Asking which system defines it has no answer, because
|
||||
every system that uses it would be a candidate.
|
||||
3. hasSource on Metrics recomputed from the elements that implement them.
|
||||
It said "SODH back-doc v0.7" -- the provenance of the record, not the
|
||||
source of the data. Asserted rather than derived, by exception, because
|
||||
"where can I get this metric" is a first-order question and the
|
||||
derivation chain will have gaps until harvesting; a shape then checks
|
||||
the assertion against the elements.
|
||||
4. The SODH Data Product, its contract and its two interfaces.
|
||||
|
||||
WHY THE PRODUCT OWNER IS EMPTY
|
||||
Nobody is appointed. The field stays absent and the shape reports a
|
||||
warning rather than a violation. Inventing a name to satisfy a blocking
|
||||
rule is exactly how BR-004 produced padding in v0.6.
|
||||
"""
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
TTL = os.path.join(REPO, "instances", "sodh.ttl")
|
||||
W = 78
|
||||
|
||||
PRODUCT = "ex:DP_06_01_001"
|
||||
CONTRACT = "ex:DC_06_01_001"
|
||||
IFACE_WH = "ex:DI_06_01_001"
|
||||
IFACE_FILE = "ex:DI_06_01_002"
|
||||
|
||||
|
||||
def clauses(body):
|
||||
out, buf, i, n = [], [], 0, len(body)
|
||||
while i < n:
|
||||
c = body[i]
|
||||
if c == '"':
|
||||
j = i + 1
|
||||
while j < n and body[j] != '"':
|
||||
j += 2 if body[j] == "\\" else 1
|
||||
j = min(j + 1, n)
|
||||
buf.append(body[i:j]); i = j; continue
|
||||
if c == ";":
|
||||
out.append("".join(buf)); buf = []; i += 1; continue
|
||||
buf.append(c); i += 1
|
||||
out.append("".join(buf))
|
||||
return out
|
||||
|
||||
|
||||
def set_prop(text, subject, prop, value):
|
||||
pat = re.compile(r'(^%s a pr:\w+ ;)(.*?)(\.\s*\n)' % re.escape(subject), re.S | re.M)
|
||||
m = pat.search(text)
|
||||
if not m:
|
||||
return text, False
|
||||
kept = [c for c in clauses(m.group(2))
|
||||
if c.strip() and not re.match(r'\s*%s\s' % re.escape(prop), c)]
|
||||
if value is not None:
|
||||
kept.insert(0, "\n %s %s " % (prop, value))
|
||||
body = " ;".join(kept).rstrip() + "\n "
|
||||
return text[:m.start()] + m.group(1) + body + m.group(3) + text[m.end():], True
|
||||
|
||||
|
||||
def get_prop(text, subject, prop):
|
||||
m = re.search(r'^%s a pr:\w+ ;(.*?)\.\s*\n' % re.escape(subject), text, re.S | re.M)
|
||||
if not m:
|
||||
return None
|
||||
for c in clauses(m.group(1)):
|
||||
mm = re.match(r'\s*%s\s+(.*)$' % re.escape(prop), c, re.S)
|
||||
if mm:
|
||||
return mm.group(1).strip()
|
||||
return None
|
||||
|
||||
|
||||
def product_blocks(objects):
|
||||
packages = " ,\n ".join(objects)
|
||||
return """# --- data product ------------------------------------------------------
|
||||
# The unit of delivery: what carries a version, an owner, a changelog and a
|
||||
# list of consumers to warn. It has no single technical counterpart on purpose
|
||||
# -- rename the schema and the product is unchanged; change its SLA and it is
|
||||
# not. Its physical footprint is derivable through packages, never asserted.
|
||||
|
||||
%s a pr:DataProduct ;
|
||||
pr:hasIdentifier "DP-06.01-001" ; pr:hasName "Sell Out Data Hub - Gold" ;
|
||||
pr:owningDomain ex:DD_06 ;
|
||||
pr:operatedBy ex:SD_06_01 ;
|
||||
pr:packages %s ;
|
||||
pr:governedBy %s ;
|
||||
pr:exposes %s , %s ;
|
||||
pr:hasStatus "DRAFT" ; pr:hasVersion "1.2" ; pr:hasSource "SODH Gold" .
|
||||
|
||||
%s a pr:DataContract ;
|
||||
pr:hasIdentifier "DC-06.01-001" ; pr:hasName "Sell Out Data Hub - Gold service commitment" ;
|
||||
pr:owningDomain ex:DD_06 ;
|
||||
pr:hasBusinessRule "Refreshed weekly, available Tuesday 06:00 CET. Panel coverage published with every load. Schema stable within a minor version; 60 days notice on any breaking change. Support through the Sales Performance data team." ;
|
||||
pr:hasStatus "DRAFT" ; pr:hasVersion "1.2" .
|
||||
|
||||
%s a pr:DataInterface ;
|
||||
pr:hasIdentifier "DI-06.01-001" ; pr:hasName "SODH Gold consumption schema" ;
|
||||
pr:owningDomain ex:DD_06 ;
|
||||
pr:hasBusinessDefinition "Read access to the Gold layer through Snowflake grants. The channel BI tools and analysts consume."@en ;
|
||||
pr:hasStatus "DRAFT" ; pr:hasVersion "1.2" ; pr:hasSource "SODH Gold" .
|
||||
|
||||
%s a pr:DataInterface ;
|
||||
pr:hasIdentifier "DI-06.01-002" ; pr:hasName "Panel partner weekly extract" ;
|
||||
pr:owningDomain ex:DD_06 ;
|
||||
pr:hasBusinessDefinition "File extract delivered to the panel provider each week. Outside Snowflake, and the boundary where lineage leaves the warehouse."@en ;
|
||||
pr:hasStatus "DRAFT" ; pr:hasVersion "1.2" ; pr:hasSource "SODH Gold" .
|
||||
""" % (PRODUCT, packages, CONTRACT, IFACE_WH, IFACE_FILE,
|
||||
CONTRACT, IFACE_WH, IFACE_FILE)
|
||||
|
||||
|
||||
def main():
|
||||
apply_changes = "--apply" in sys.argv
|
||||
if not os.path.exists(TTL):
|
||||
print("Not found: %s" % TTL); sys.exit(2)
|
||||
text = original = open(TTL, encoding="utf-8").read()
|
||||
report = []
|
||||
|
||||
# 1. term rename
|
||||
text, n = re.subn(r'\bpr:physicalizedIn\b', 'pr:storedIn', text)
|
||||
report.append("physicalizedIn -> storedIn: %d occurrence(s)" % n)
|
||||
|
||||
# 2. no source of truth on a concept
|
||||
n_bc = 0
|
||||
for m in list(re.finditer(r'^(ex:BC_\w+) a pr:BusinessConcept ;', text, re.M)):
|
||||
if get_prop(text, m.group(1), "pr:hasSource") is not None:
|
||||
text, _ = set_prop(text, m.group(1), "pr:hasSource", None)
|
||||
n_bc += 1
|
||||
report.append("hasSource removed from %d business concepts" % n_bc)
|
||||
|
||||
# 3. metric source of truth, from the elements that implement it
|
||||
de_src = {}
|
||||
for m in re.finditer(r'^(ex:DE_\w+) a pr:DataElement ;(.*?)\.\s*\n', text, re.S | re.M):
|
||||
v = re.search(r'pr:hasSource\s+"([^"]*)"', m.group(2))
|
||||
if v:
|
||||
de_src[m.group(1)] = v.group(1)
|
||||
wired = {}
|
||||
for m in re.finditer(r'(ex:M_\w+)\s+pr:computedBy(.*?)\.', text, re.S):
|
||||
wired[m.group(1)] = re.findall(r'ex:DE_\w+', m.group(2))
|
||||
n_m, mixed = 0, []
|
||||
for metric, des in wired.items():
|
||||
srcs = sorted({de_src[d] for d in des if d in de_src})
|
||||
if not srcs:
|
||||
continue
|
||||
if len(srcs) > 1:
|
||||
mixed.append(metric)
|
||||
text, ok = set_prop(text, metric, "pr:hasSource",
|
||||
" , ".join('"%s"' % x for x in srcs))
|
||||
n_m += ok
|
||||
report.append("source of truth recomputed on %d metrics%s"
|
||||
% (n_m, "" if not mixed else
|
||||
" (%d blend several systems)" % len(mixed)))
|
||||
|
||||
# 4. the data product
|
||||
if not re.search(r'^%s a ' % re.escape(PRODUCT), text, re.M):
|
||||
objs = re.findall(r'^(ex:DO_\w+) a pr:DataObject ;', text, re.M)
|
||||
text = text.rstrip() + "\n\n\n" + product_blocks(objs)
|
||||
report.append("data product created, packaging %d data objects" % len(objs))
|
||||
else:
|
||||
report.append("data product already present, skipped")
|
||||
|
||||
# ---- report
|
||||
print()
|
||||
print("SODH MIGRATION v1.1 -> v1.2 %s" % ("APPLY" if apply_changes else "DRY RUN"))
|
||||
print("=" * W)
|
||||
for line in report:
|
||||
print(" " + line)
|
||||
print("=" * W)
|
||||
bc_src = len(re.findall(r'^ex:BC_\w+ a pr:BusinessConcept ;(?:(?!\.\s*\n).)*?hasSource',
|
||||
text, re.S | re.M))
|
||||
defined = set(re.findall(r'^(ex:\w+) a ', text, re.M))
|
||||
dangling = sorted(set(re.findall(r'\bex:\w+', text)) - defined)
|
||||
doubled = []
|
||||
for m in re.finditer(r'^(ex:\w+) a pr:\w+ ;(.*?)\.\s*\n', text, re.S | re.M):
|
||||
for p in ("pr:hasName", "pr:hasSource", "pr:operatedBy", "pr:governedBy"):
|
||||
if len(re.findall(r'(?<![\w:])%s\s' % re.escape(p), m.group(2))) > 1:
|
||||
doubled.append("%s/%s" % (m.group(1), p))
|
||||
print(" concepts still carrying a source: %d (must be 0)" % bc_src)
|
||||
print(" physicalizedIn left: %d (must be 0)"
|
||||
% len(re.findall(r'pr:physicalizedIn', text)))
|
||||
print(" dangling references: %s" % (", ".join(dangling) if dangling else "none"))
|
||||
print(" duplicated single-valued properties: %s"
|
||||
% (", ".join(doubled) if doubled else "none"))
|
||||
print("=" * W)
|
||||
|
||||
if text != original and apply_changes:
|
||||
shutil.copy2(TTL, TTL + ".bak")
|
||||
open(TTL, "w", encoding="utf-8").write(text)
|
||||
print(" written, backup at %s.bak" % os.path.basename(TTL))
|
||||
elif text != original:
|
||||
print(" dry run -- re-run with --apply to write")
|
||||
print()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,312 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
PR Data Meta Model - SHACL validation runner
|
||||
=============================================
|
||||
The automated gate behind rule DQ-011. Validates an instance graph against the
|
||||
executable rulebook (SHACL shapes), with the T-Box supplying subclass
|
||||
entailment.
|
||||
|
||||
USAGE
|
||||
python3 scripts/run_shacl_validation.py # all instances
|
||||
python3 scripts/run_shacl_validation.py instances/sodh.ttl # one file
|
||||
python3 scripts/run_shacl_validation.py --quiet # CI mode
|
||||
|
||||
Paths resolve against the repository root (the parent of scripts/), so the
|
||||
script runs correctly from anywhere.
|
||||
|
||||
EXIT CODES
|
||||
0 conforms, no blocking violation -> eligible for PUBLISHED
|
||||
1 violations found -> stays DRAFT / UNDER_REVIEW
|
||||
2 the run itself is not trustworthy -> a green result would be meaningless
|
||||
|
||||
Exit code 2 is the important one. A validation gate that passes because no
|
||||
shape matched anything is worse than one that fails: it reports success while
|
||||
checking nothing. The pre-flight below refuses to let that happen.
|
||||
|
||||
DEPENDENCIES
|
||||
pyshacl, rdflib. On GrosseBertha these need a venv built with the STABLE
|
||||
python (never the DSM system python, which moves between DSM releases):
|
||||
|
||||
/usr/local/bin/python3.9 -m venv venv
|
||||
. venv/bin/activate
|
||||
pip install pyshacl rdflib
|
||||
"""
|
||||
import glob
|
||||
import os
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
ONTOLOGY = os.path.join(REPO, "ontology", "pr_metamodel.ttl")
|
||||
SHAPES = os.path.join(REPO, "shapes", "pr_metamodel_shapes.ttl")
|
||||
INSTANCES = os.path.join(REPO, "instances")
|
||||
|
||||
W = 78
|
||||
BAR = "=" * W
|
||||
|
||||
|
||||
def rel(path):
|
||||
"""Path relative to the repo root, for readable output."""
|
||||
try:
|
||||
return os.path.relpath(path, REPO)
|
||||
except ValueError:
|
||||
return path
|
||||
|
||||
|
||||
def die(code, title, lines):
|
||||
print(BAR)
|
||||
print("PR META MODEL - SHACL VALIDATION")
|
||||
print(BAR)
|
||||
print(title)
|
||||
print("-" * W)
|
||||
for line in lines:
|
||||
print(" " + line)
|
||||
print(BAR)
|
||||
sys.exit(code)
|
||||
|
||||
|
||||
def load_deps():
|
||||
"""
|
||||
Import the validation stack.
|
||||
|
||||
Catches every exception, not just ImportError: a dependency that installs
|
||||
but cannot be imported (pyshacl built for a newer Python than the venv,
|
||||
an rdflib major-version clash) raises TypeError or AttributeError at import
|
||||
time. Letting those escape would end the process with an exit code CI reads
|
||||
as "the model has violations", when in fact validation never started.
|
||||
Anything wrong here is exit 2 -- the run is not trustworthy.
|
||||
"""
|
||||
try:
|
||||
from pyshacl import validate
|
||||
from rdflib import Graph
|
||||
from rdflib.namespace import RDF, SH
|
||||
return validate, Graph, RDF, SH
|
||||
except Exception as exc:
|
||||
hint = []
|
||||
if "unsupported operand type" in str(exc) or "|" in str(exc):
|
||||
hint = [
|
||||
"This is PEP 604 syntax (str | X) evaluated at import time:",
|
||||
"the installed pyshacl needs Python 3.10+, this venv is %d.%d."
|
||||
% (sys.version_info[0], sys.version_info[1]),
|
||||
"pyshacl advertises 3.9 support but its code does not honour it,",
|
||||
"so pip installs a version that cannot be imported.",
|
||||
"",
|
||||
"Pin a release that really supports this interpreter:",
|
||||
' pip install "pyshacl==0.26.0" "rdflib>=6.3.2,<7.1"',
|
||||
"or rebuild the venv on a newer Python if one exists.",
|
||||
]
|
||||
else:
|
||||
hint = [
|
||||
"Build the venv with the stable python, never the DSM system one:",
|
||||
" /usr/local/bin/python3.9 -m venv venv",
|
||||
" . venv/bin/activate && pip install pyshacl rdflib",
|
||||
]
|
||||
die(2, "DEPENDENCY UNUSABLE -- validation did not run",
|
||||
["%s: %s" % (type(exc).__name__, exc), ""] + hint)
|
||||
|
||||
|
||||
def term_namespaces(graph, predicate=None):
|
||||
"""
|
||||
Namespaces actually used by pr: terms in a graph, with usage counts.
|
||||
|
||||
Only real IRIs count. Literals are skipped: SPARQL constraint bodies
|
||||
(sh:select) embed their own PREFIX declarations, and counting those would
|
||||
put raw query text in the diagnostic table.
|
||||
"""
|
||||
counts = defaultdict(int)
|
||||
for s, p, o in graph:
|
||||
for term in (s, p, o):
|
||||
text = str(term)
|
||||
if "ontology.pernod-ricard.com/metamodel" not in text:
|
||||
continue
|
||||
if not text.startswith("http") or any(c in text for c in " \t\n\r<>\""):
|
||||
continue # literal, not an IRI
|
||||
counts[text.rsplit("/", 1)[0] + "/"] += 1
|
||||
return counts
|
||||
|
||||
|
||||
def preflight(data, shapes, onto, data_files):
|
||||
"""
|
||||
Refuse to produce a green result that means nothing.
|
||||
|
||||
Two ways a SHACL run passes while checking nothing:
|
||||
1. the shapes target classes in a namespace the instances never use;
|
||||
2. the ontology declares its classes in a third namespace, so subclass
|
||||
entailment cannot reach the instances and every shape targeting an
|
||||
abstract parent (MetaModelObject) finds zero focus nodes.
|
||||
Both look identical to a caller reading only "CONFORMS: True".
|
||||
"""
|
||||
problems = []
|
||||
ns_problem = False
|
||||
|
||||
if not len(data):
|
||||
problems.append("The instance graph is empty -- nothing to validate.")
|
||||
|
||||
ns_data = term_namespaces(data)
|
||||
ns_shapes = term_namespaces(shapes)
|
||||
ns_onto = term_namespaces(onto) if onto is not None else {}
|
||||
|
||||
if ns_data and ns_shapes and not (set(ns_data) & set(ns_shapes)):
|
||||
ns_problem = True
|
||||
problems.append(
|
||||
"Shapes and instances use disjoint term namespaces -- no shape can "
|
||||
"match any node.")
|
||||
|
||||
if ns_onto and ns_data and not (set(ns_onto) & set(ns_data)):
|
||||
ns_problem = True
|
||||
problems.append(
|
||||
"The T-Box declares its terms in a namespace the instances never "
|
||||
"use. Subclass entailment cannot reach the data, so every shape "
|
||||
"targeting an abstract parent finds zero focus nodes.")
|
||||
|
||||
if problems:
|
||||
lines = list(problems)
|
||||
lines.append("")
|
||||
lines.append("Namespaces in use:")
|
||||
for label, counts in (("instances", ns_data), ("shapes", ns_shapes),
|
||||
("ontology", ns_onto)):
|
||||
if counts:
|
||||
for ns, n in sorted(counts.items(), key=lambda kv: -kv[1]):
|
||||
lines.append(" %-10s %-52s %6d refs" % (label, ns, n))
|
||||
else:
|
||||
lines.append(" %-10s (none)" % label)
|
||||
if ns_problem:
|
||||
lines.append("")
|
||||
lines.append("Fix: align the namespaces before trusting any result.")
|
||||
lines.append(" sed -i 's|/metamodel/0.9/|/metamodel/|g' "
|
||||
"instances/*.ttl shapes/*.ttl")
|
||||
die(2, "PRE-FLIGHT FAILED -- a result here would be meaningless", lines)
|
||||
|
||||
|
||||
def main():
|
||||
args = [a for a in sys.argv[1:] if not a.startswith("-")]
|
||||
quiet = "--quiet" in sys.argv or "-q" in sys.argv
|
||||
|
||||
validate, Graph, RDF, SH = load_deps()
|
||||
|
||||
# ---- resolve inputs ---------------------------------------------------
|
||||
if args:
|
||||
data_files = [a if os.path.isabs(a) else os.path.join(REPO, a) for a in args]
|
||||
else:
|
||||
data_files = sorted(glob.glob(os.path.join(INSTANCES, "*.ttl")))
|
||||
|
||||
missing = [f for f in data_files if not os.path.exists(f)]
|
||||
if missing or not data_files:
|
||||
die(2, "NO INSTANCE DATA -- validation did not run",
|
||||
[rel(f) + " not found" for f in missing] or
|
||||
["instances/ contains no .ttl file"])
|
||||
|
||||
if not os.path.exists(SHAPES):
|
||||
die(2, "NO SHAPES -- validation did not run",
|
||||
[rel(SHAPES) + " not found"])
|
||||
|
||||
# ---- parse ------------------------------------------------------------
|
||||
data = Graph()
|
||||
for f in data_files:
|
||||
try:
|
||||
data.parse(f, format="turtle")
|
||||
except Exception as exc:
|
||||
die(2, "PARSE ERROR -- validation did not run",
|
||||
[rel(f), str(exc)])
|
||||
|
||||
try:
|
||||
shapes = Graph().parse(SHAPES, format="turtle")
|
||||
except Exception as exc:
|
||||
die(2, "PARSE ERROR in shapes -- validation did not run",
|
||||
[rel(SHAPES), str(exc)])
|
||||
|
||||
onto = None
|
||||
if os.path.exists(ONTOLOGY):
|
||||
try:
|
||||
onto = Graph().parse(ONTOLOGY, format="turtle")
|
||||
except Exception as exc:
|
||||
die(2, "PARSE ERROR in ontology -- validation did not run",
|
||||
[rel(ONTOLOGY), str(exc)])
|
||||
|
||||
preflight(data, shapes, onto, data_files)
|
||||
|
||||
# ---- validate ---------------------------------------------------------
|
||||
conforms, results, _ = validate(
|
||||
data_graph=data,
|
||||
shacl_graph=shapes,
|
||||
ont_graph=onto,
|
||||
inference="rdfs", # a Metric is also a MetaModelObject
|
||||
advanced=True, # SPARQL constraints (NR-002, OW-007, DQ-006)
|
||||
abort_on_first=False,
|
||||
meta_shacl=False,
|
||||
)
|
||||
|
||||
def shape_label(g, res, SH):
|
||||
"""
|
||||
A readable name for the shape that fired.
|
||||
|
||||
sh:sourceShape on a property constraint is a blank node, which prints as
|
||||
'ne775aabb...' and tells the reader nothing. Fall back, in order, to the
|
||||
constrained path, then to the named NodeShape that owns the blank node,
|
||||
then to the constraint component.
|
||||
"""
|
||||
shape = g.value(res, SH.sourceShape)
|
||||
if shape is not None and not str(shape).startswith("n"):
|
||||
name = str(shape).rsplit("/", 1)[-1]
|
||||
if name and not name.startswith("N"):
|
||||
return name
|
||||
path = g.value(res, SH.resultPath)
|
||||
if path is not None:
|
||||
owner = None
|
||||
for s_, p_, o_ in g.triples((None, SH.property, shape)):
|
||||
owner = s_
|
||||
break
|
||||
base = str(path).rsplit("/", 1)[-1].rsplit("#", 1)[-1]
|
||||
if owner is not None and not str(owner).startswith("n"):
|
||||
return "%s / %s" % (str(owner).rsplit("/", 1)[-1], base)
|
||||
return "path %s" % base
|
||||
comp = g.value(res, SH.sourceConstraintComponent)
|
||||
if comp is not None:
|
||||
return str(comp).rsplit("#", 1)[-1]
|
||||
return "(unnamed shape)"
|
||||
|
||||
rows = []
|
||||
for res in results.subjects(RDF.type, SH.ValidationResult):
|
||||
sev = str(results.value(res, SH.resultSeverity)).rsplit("#", 1)[-1]
|
||||
focus = str(results.value(res, SH.focusNode)).rsplit("/", 1)[-1]
|
||||
msg = str(results.value(res, SH.resultMessage) or "")
|
||||
rows.append((sev, shape_label(results, res, SH), focus, msg))
|
||||
|
||||
violations = sum(1 for r in rows if r[0] == "Violation")
|
||||
warnings = len(rows) - violations
|
||||
|
||||
# ---- report -----------------------------------------------------------
|
||||
print(BAR)
|
||||
print("PR META MODEL - SHACL VALIDATION")
|
||||
print(" data : " + ", ".join(rel(f) for f in data_files))
|
||||
print(" shapes : " + rel(SHAPES))
|
||||
print(" ontology : " + (rel(ONTOLOGY) if onto is not None else "(absent - no entailment)"))
|
||||
print(" triples : %d data / %d shapes" % (len(data), len(shapes)))
|
||||
print(BAR)
|
||||
print("CONFORMS : %s" % conforms)
|
||||
print("RESULTS : %d total | %d BLOCKING | %d MAJOR (warning)"
|
||||
% (len(rows), violations, warnings))
|
||||
|
||||
if rows and not quiet:
|
||||
print("-" * W)
|
||||
by_shape = defaultdict(list)
|
||||
for sev, src, focus, msg in rows:
|
||||
by_shape[(sev, src)].append((focus, msg))
|
||||
# blocking first, then by volume: the rules to fix first come first
|
||||
order = sorted(by_shape.items(),
|
||||
key=lambda kv: (kv[0][0] != "Violation", -len(kv[1])))
|
||||
for (sev, src), items in order:
|
||||
tag = "BLOCK" if sev == "Violation" else "WARN "
|
||||
print("[%s] %-42s %3d node(s)" % (tag, src or "(unnamed shape)", len(items)))
|
||||
print(" %s" % (items[0][1][:66] or "(no message)"))
|
||||
for focus, _ in items[:6]:
|
||||
print(" - %s" % focus)
|
||||
if len(items) > 6:
|
||||
print(" ... and %d more" % (len(items) - 6))
|
||||
|
||||
print(BAR)
|
||||
sys.exit(0 if violations == 0 else 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+472
-109
@@ -1,127 +1,490 @@
|
||||
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
|
||||
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
|
||||
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
|
||||
@prefix sh: <http://www.w3.org/ns/shacl#> .
|
||||
@prefix pr: <https://ontology.pernod-ricard.com/metamodel/0.9/> .
|
||||
@prefix prsh: <https://ontology.pernod-ricard.com/metamodel/0.9/shapes/> .
|
||||
@prefix sh: <http://www.w3.org/ns/shacl#> .
|
||||
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
|
||||
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
|
||||
@prefix dcterms: <http://purl.org/dc/terms/> .
|
||||
@prefix pr: <https://ontology.pernod-ricard.com/metamodel/> .
|
||||
@prefix prs: <https://ontology.pernod-ricard.com/metamodel/shapes/> .
|
||||
|
||||
#################################################################
|
||||
# PR Data Meta Model - SHACL Shapes v0.1.1
|
||||
# Executable encoding of BLOCKING (sh:Violation) & MAJOR (sh:Warning) rules.
|
||||
# FIX v0.1.1 (2026-07-23): NR-002 uniqueness query now restricts ?cls to
|
||||
# LEAF classes (via FILTER ... IN) so that objects of different layers that
|
||||
# share a name (e.g. Sub-Domain "Sell Out" vs Concept "Sell Out") are NOT
|
||||
# reported as duplicates. Rule stays "unique WITHIN its class".
|
||||
# Run: python run_shacl_validation.py SODH_data.ttl pr_metamodel_shapes_v0.1.ttl pr_metamodel_v0.9.ttl
|
||||
# PR DATA META MODEL - SHACL SHAPES v0.2
|
||||
# Aligned with T-Box v1.1 and Rulebook v0.3.x
|
||||
#
|
||||
# WHAT CHANGED vs v0.1.1
|
||||
# ----------------------
|
||||
# 1. SPLIT BY PROVENANCE. v0.1.1 targeted pr:MetaModelObject and demanded an
|
||||
# identifier, a name, an owner, an owning domain, a status and a version on
|
||||
# everything. Applied to the harvested half of the graph that would fail on
|
||||
# every one of ~50 000 Snowflake columns. Curated and Observed objects now
|
||||
# have separate shapes with separate expectations.
|
||||
#
|
||||
# 2. SCOPE FOR THE UNIFIED RELATIONS. belongsTo and represents deliberately
|
||||
# carry no rdfs:domain/range in the T-Box, because each serves two classes
|
||||
# with different targets. The scope lives here instead, per class.
|
||||
#
|
||||
# 3. MEANING IS AN XOR, NOT A CHECKLIST. A Data Element reaches business
|
||||
# meaning by exactly one route. sh:xone rejects both zero routes and two.
|
||||
#
|
||||
# 4. NO COVERAGE RULES. Nothing here requires a Business Concept to be
|
||||
# realized, or a Business Object to hold N concepts. That was BR-004, and
|
||||
# encoding a coverage expectation as a blocking constraint does not produce
|
||||
# coverage -- it produces padding. Coverage is a dashboard metric.
|
||||
#
|
||||
# 5. PORTABLE REGEX. No lookbehind: sh:pattern is normatively XPath 2.0
|
||||
# regex, which does not support it. Written as sh:not [ sh:pattern ... ].
|
||||
#
|
||||
# SEVERITY MAPPING
|
||||
# sh:Violation -> BLOCKING, cannot reach PUBLISHED, fails CI (DQ-011)
|
||||
# sh:Warning -> MAJOR, must be reviewed, does not block
|
||||
# sh:Info -> GUIDELINE, informational
|
||||
#################################################################
|
||||
|
||||
prsh:MetaModelObjectShape a sh:NodeShape ;
|
||||
sh:targetClass pr:MetaModelObject ;
|
||||
sh:property [ sh:path pr:hasIdentifier ; sh:minCount 1 ; sh:maxCount 1 ; sh:datatype xsd:string ;
|
||||
sh:severity sh:Violation ; sh:message "DQ-001 (BLOCKING): exactly one hasIdentifier." ] ;
|
||||
sh:property [ sh:path pr:hasName ; sh:minCount 1 ; sh:maxCount 1 ; sh:datatype xsd:string ; sh:minLength 2 ;
|
||||
sh:severity sh:Violation ; sh:message "DQ-002 (BLOCKING): exactly one non-empty hasName." ] ;
|
||||
sh:property [ sh:path pr:ownedBy ; sh:minCount 1 ; sh:class pr:Actor ;
|
||||
sh:severity sh:Violation ; sh:message "OW-001 (BLOCKING): at least one owner (Actor)." ] ;
|
||||
sh:property [ sh:path pr:owningDomain ; sh:minCount 1 ; sh:maxCount 1 ; sh:class pr:DataDomain ;
|
||||
sh:severity sh:Violation ; sh:message "OW-001 (BLOCKING): exactly one owning Data Domain." ] ;
|
||||
sh:property [ sh:path pr:hasStatus ; sh:minCount 1 ; sh:in ( "DRAFT" "UNDER_REVIEW" "PUBLISHED" "DEPRECATED" ) ;
|
||||
sh:severity sh:Warning ; sh:message "DQ-012 (MAJOR): hasStatus in controlled list." ] ;
|
||||
sh:property [ sh:path pr:hasVersion ; sh:minCount 1 ;
|
||||
sh:severity sh:Warning ; sh:message "DQ-012 (MAJOR): hasVersion required." ] .
|
||||
<https://ontology.pernod-ricard.com/metamodel/shapes/>
|
||||
dcterms:title "PR Data Meta Model - SHACL Shapes" ;
|
||||
dcterms:modified "2026-07-27"^^xsd:date ;
|
||||
rdfs:comment "Executable form of the rulebook. Validated by scripts/run_shacl_validation.py." .
|
||||
|
||||
prsh:BackDocSourceShape a sh:NodeShape ;
|
||||
sh:targetClass pr:MetaModelObject ;
|
||||
sh:sparql [ sh:severity sh:Violation ;
|
||||
sh:message "DQ-006 (BLOCKING): a DRAFT back-doc object MUST carry hasSource." ;
|
||||
sh:select """
|
||||
PREFIX pr: <https://ontology.pernod-ricard.com/metamodel/0.9/>
|
||||
SELECT $this WHERE { $this pr:hasStatus "DRAFT" .
|
||||
FILTER NOT EXISTS { $this pr:hasSource ?s } }
|
||||
""" ] .
|
||||
|
||||
prsh:CanonicalNameShape a sh:NodeShape ;
|
||||
sh:targetClass pr:MetaModelObject ;
|
||||
sh:property [ sh:path pr:hasName ;
|
||||
sh:pattern "^(?!GL_)(?!ID_).*(?<!_NM)(?<!_CD)(?<!_FLG)(?<!_NR)(?<!_DT)$" ; sh:flags "i" ;
|
||||
#################################################################
|
||||
# IDENTITY AND LIFECYCLE -- curated objects only
|
||||
#################################################################
|
||||
|
||||
prs:CuratedObjectShape a sh:NodeShape ;
|
||||
sh:targetClass pr:CuratedObject ;
|
||||
rdfs:comment "Applies to everything governance asserts. Observed objects are deliberately exempt: see ObservedObjectShape." ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:hasIdentifier ; sh:datatype xsd:string ;
|
||||
sh:minCount 1 ; sh:maxCount 1 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "DQ-001 -- every curated object carries exactly one identifier. NOTE: uniqueness across the graph is NOT tested here; SHACL checks cardinality only. The reconciliation job tests uniqueness." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:hasName ; sh:datatype xsd:string ;
|
||||
sh:minCount 1 ; sh:maxCount 1 ; sh:minLength 3 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "Every curated object carries exactly one canonical name." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:hasStatus ; sh:minCount 1 ;
|
||||
sh:in ( "DRAFT" "UNDER_REVIEW" "PUBLISHED" "DEPRECATED" "RETIRED" ) ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "LC-001 -- status must be one of DRAFT, UNDER_REVIEW, PUBLISHED, DEPRECATED, RETIRED." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:owningDomain ; sh:maxCount 1 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "OW-001 -- at most one owning Data Domain. On a Data Element this may be re-declared to differ from the parent chain: that marks a borrowed element." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:hasVersion ; sh:maxCount 1 ;
|
||||
sh:severity sh:Warning ;
|
||||
sh:message "NR-001 (MAJOR): canonical name free of physical prefixes/suffixes." ] .
|
||||
sh:message "A published object should carry a version." ] .
|
||||
|
||||
prsh:MetricShape a sh:NodeShape ;
|
||||
sh:targetClass pr:Metric ;
|
||||
sh:property [ sh:path pr:hasFormula ; sh:minCount 1 ; sh:datatype xsd:string ; sh:minLength 2 ;
|
||||
sh:severity sh:Violation ; sh:message "BR-003 / DQ-004 (BLOCKING): a Metric MUST have a formula." ] ;
|
||||
sh:property [ sh:path pr:hasUnit ; sh:minCount 1 ;
|
||||
sh:severity sh:Violation ; sh:message "DQ-004 (BLOCKING): a Metric MUST have a hasUnit." ] ;
|
||||
sh:property [ sh:path pr:hasGranularity ; sh:minCount 1 ;
|
||||
sh:severity sh:Violation ; sh:message "DQ-004 (BLOCKING): a Metric MUST have a hasGranularity." ] .
|
||||
|
||||
prsh:BusinessConceptShape a sh:NodeShape ;
|
||||
sh:targetClass pr:BusinessConcept ;
|
||||
sh:property [ sh:path pr:hasFormula ; sh:maxCount 0 ;
|
||||
sh:severity sh:Violation ; sh:message "BR-003 (BLOCKING): a Business Concept MUST NOT have a formula." ] .
|
||||
prs:PublishedShape a sh:NodeShape ;
|
||||
sh:targetClass pr:CuratedObject ;
|
||||
rdfs:comment "Extra demands that only bite once an object claims to be PUBLISHED. Keeping these off the base shape is what lets a domain be documented progressively." ;
|
||||
|
||||
prsh:BusinessObjectShape a sh:NodeShape ;
|
||||
sh:targetClass pr:BusinessObject ;
|
||||
sh:property [ sh:path pr:composedOf ; sh:minCount 3 ; sh:maxCount 10 ; sh:class pr:BusinessConcept ;
|
||||
sh:severity sh:Warning ; sh:message "BR-004 (MAJOR): a Business Object MUST group 3 to 10 Business Concepts." ] ;
|
||||
sh:property [ sh:path pr:belongsToSubDomain ; sh:minCount 1 ; sh:maxCount 1 ; sh:class pr:SubDomain ;
|
||||
sh:severity sh:Warning ; sh:message "NR-012/OW (MAJOR): a Business Object belongs to exactly one Sub-Domain." ] ;
|
||||
sh:property [ sh:path pr:hasSteward ; sh:minCount 1 ; sh:class pr:DataSteward ;
|
||||
sh:severity sh:Warning ; sh:message "OW-005 (MAJOR): a Business Object MUST have a Data Steward." ] .
|
||||
sh:sparql [
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "OW-007 -- an object flagged TO_ARBITRATE cannot be PUBLISHED. Resolve at the Cross-Domain Council first." ;
|
||||
sh:select """
|
||||
PREFIX pr: <https://ontology.pernod-ricard.com/metamodel/>
|
||||
SELECT $this WHERE {
|
||||
$this pr:hasStatus "PUBLISHED" ;
|
||||
pr:arbitrationStatus "TO_ARBITRATE" .
|
||||
}""" ] ;
|
||||
|
||||
prsh:DataObjectShape a sh:NodeShape ;
|
||||
sh:targetClass pr:DataObject ;
|
||||
sh:property [ sh:path pr:implements ; sh:minCount 1 ; sh:maxCount 1 ; sh:class pr:BusinessObject ;
|
||||
sh:severity sh:Violation ; sh:message "BR-006 (BLOCKING): a Data Object MUST implement exactly one Business Object." ] ;
|
||||
sh:property [ sh:path pr:hasElement ; sh:minCount 1 ; sh:class pr:DataElement ;
|
||||
sh:severity sh:Violation ; sh:message "BR-006 (BLOCKING): a Data Object MUST contain at least one Data Element." ] ;
|
||||
sh:property [ sh:path pr:hasSteward ; sh:minCount 1 ; sh:class pr:DataSteward ;
|
||||
sh:severity sh:Warning ; sh:message "OW-005 (MAJOR): a Data Object MUST have a Data Steward." ] .
|
||||
sh:sparql [
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "OW-003 -- a PUBLISHED object must have an owning Data Domain." ;
|
||||
sh:select """
|
||||
PREFIX pr: <https://ontology.pernod-ricard.com/metamodel/>
|
||||
SELECT $this WHERE {
|
||||
$this pr:hasStatus "PUBLISHED" .
|
||||
FILTER NOT EXISTS { $this pr:owningDomain ?d }
|
||||
}""" ] .
|
||||
|
||||
prsh:DataElementShape a sh:NodeShape ;
|
||||
sh:targetClass pr:DataElement ;
|
||||
sh:property [ sh:path pr:hasFormat ; sh:minCount 1 ;
|
||||
sh:severity sh:Warning ; sh:message "DQ-005 (MAJOR): a Data Element MUST have a hasFormat." ] ;
|
||||
sh:property [ sh:path pr:hasSourceSystem ; sh:minCount 1 ;
|
||||
sh:severity sh:Warning ; sh:message "DQ-005 (MAJOR): a Data Element MUST have a hasSourceSystem." ] ;
|
||||
sh:property [ sh:path pr:physicalName ; sh:minCount 1 ; sh:maxCount 1 ; sh:datatype xsd:string ;
|
||||
sh:severity sh:Violation ; sh:message "DQ-010 (BLOCKING): a mapped Data Element MUST have a physical_name." ] .
|
||||
|
||||
prsh:DataDomainShape a sh:NodeShape ;
|
||||
#################################################################
|
||||
# OBSERVED OBJECTS -- harvested, never hand-edited
|
||||
#################################################################
|
||||
|
||||
prs:ObservedObjectShape a sh:NodeShape ;
|
||||
sh:targetClass pr:ObservedObject ;
|
||||
rdfs:comment "No identifier, no name, no owner, no status, no version. A harvested column is attributed through the Data Element mapped to it, never asserted directly. What it must carry is enough to be re-found on the next run." ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:sourceIdentifier ; sh:datatype xsd:string ;
|
||||
sh:maxCount 1 ;
|
||||
sh:severity sh:Warning ;
|
||||
sh:message "A native immutable source id makes reconciliation idempotent and survives renames, which names do not." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:harvestedOn ; sh:datatype xsd:dateTime ;
|
||||
sh:minCount 1 ; sh:maxCount 1 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "Every observed object records when it was harvested. Without it there is no way to tell a current fact from a stale one." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:owningDomain ; sh:maxCount 0 ;
|
||||
sh:severity sh:Warning ;
|
||||
sh:message "Observed objects are not attributed directly -- ownership flows from the Data Element mapped to them." ] .
|
||||
|
||||
|
||||
prs:FieldShape a sh:NodeShape ;
|
||||
sh:targetClass pr:Field ;
|
||||
sh:property [
|
||||
sh:path pr:fullyQualifiedName ; sh:datatype xsd:string ;
|
||||
sh:minCount 1 ; sh:maxCount 1 ;
|
||||
sh:pattern "^[A-Za-z0-9_$]+\\\\.[A-Za-z0-9_$]+\\\\.[A-Za-z0-9_$]+\\\\.[A-Za-z0-9_$]+$" ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "A Field carries DATABASE.SCHEMA.RELATION.COLUMN. A bare column name is ambiguous as soon as it exists in two tables, so the FQN is the real join key against INFORMATION_SCHEMA." ] ;
|
||||
sh:property [
|
||||
sh:path pr:livesIn ; sh:minCount 1 ; sh:maxCount 1 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "Every Field lives in exactly one Physical Relation." ] .
|
||||
|
||||
|
||||
#################################################################
|
||||
# CATEGORIZATION TREE -- scope of the unified belongsTo
|
||||
#################################################################
|
||||
|
||||
prs:SubDomainShape a sh:NodeShape ;
|
||||
sh:targetClass pr:SubDomain ;
|
||||
sh:property [
|
||||
sh:path pr:belongsTo ; sh:class pr:DataDomain ;
|
||||
sh:minCount 1 ; sh:maxCount 1 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "A Sub-Domain belongs to exactly one Data Domain." ] ;
|
||||
sh:property [
|
||||
sh:path pr:hasSubDomainOwner ; sh:maxCount 1 ;
|
||||
sh:severity sh:Warning ;
|
||||
sh:message "A Sub-Domain should name its owner." ] .
|
||||
|
||||
|
||||
prs:DataDomainShape a sh:NodeShape ;
|
||||
sh:targetClass pr:DataDomain ;
|
||||
sh:property [ sh:path pr:hasDGL ; sh:minCount 1 ; sh:maxCount 1 ;
|
||||
sh:severity sh:Violation ; sh:message "OW-004 (BLOCKING): a Data Domain MUST have exactly one DGL." ] ;
|
||||
sh:property [ sh:path pr:hasActivationStatus ; sh:minCount 1 ; sh:maxCount 1 ;
|
||||
sh:severity sh:Violation ; sh:message "OW-004 (BLOCKING): a Data Domain MUST have exactly one Activation Status." ] .
|
||||
sh:property [
|
||||
sh:path pr:hasDomainOwner ; sh:maxCount 1 ;
|
||||
sh:severity sh:Warning ;
|
||||
sh:message "A Data Domain should name its owner." ] ;
|
||||
sh:property [
|
||||
sh:path pr:hasDGL ; sh:maxCount 1 ;
|
||||
sh:severity sh:Warning ;
|
||||
sh:message "A Data Domain should name its Data Governance Lead." ] ;
|
||||
sh:property [
|
||||
sh:path pr:hasActivationStatus ; sh:maxCount 1 ;
|
||||
sh:severity sh:Info ] .
|
||||
|
||||
# FIX v0.1.1 - NR-002 now restricts ?cls to LEAF classes so cross-layer
|
||||
# homonyms (Sub-Domain vs Concept, etc.) are not flagged as duplicates.
|
||||
prsh:NameUniquenessShape a sh:NodeShape ;
|
||||
sh:targetClass pr:MetaModelObject ;
|
||||
sh:sparql [ sh:severity sh:Violation ;
|
||||
sh:message "NR-002 (BLOCKING): canonical name MUST be unique within its (leaf) class and owning Data Domain." ;
|
||||
sh:select """
|
||||
PREFIX pr: <https://ontology.pernod-ricard.com/metamodel/0.9/>
|
||||
SELECT $this WHERE {
|
||||
$this a ?cls ; pr:hasName ?n ; pr:owningDomain ?d .
|
||||
?other a ?cls ; pr:hasName ?n ; pr:owningDomain ?d .
|
||||
FILTER ( $this != ?other )
|
||||
FILTER ( ?cls IN (
|
||||
pr:DataDomain, pr:SubDomain, pr:BusinessObject,
|
||||
pr:BusinessConcept, pr:Metric, pr:KPI,
|
||||
pr:DataAsset, pr:DataObject, pr:DataElement,
|
||||
pr:DataContract, pr:DataProduct, pr:Table, pr:Field ) )
|
||||
}
|
||||
""" ] .
|
||||
|
||||
prsh:ArbitrationGateShape a sh:NodeShape ;
|
||||
sh:targetClass pr:MetaModelObject ;
|
||||
sh:sparql [ sh:severity sh:Violation ;
|
||||
sh:message "OW-007 (BLOCKING): a TO_ARBITRATE object MUST NOT be PUBLISHED." ;
|
||||
#################################################################
|
||||
# BUSINESS OBJECT -- subject, perimeter, stewardship
|
||||
#################################################################
|
||||
|
||||
prs:BusinessObjectShape a sh:NodeShape ;
|
||||
sh:targetClass pr:BusinessObject ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:belongsTo ; sh:class pr:SubDomain ;
|
||||
sh:minCount 1 ; sh:maxCount 1 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "A Business Object belongs to exactly one Sub-Domain." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:aboutConcept ; sh:class pr:BusinessConcept ;
|
||||
sh:minCount 1 ; sh:maxCount 1 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "BR-004 -- a Business Object is about exactly one Business Concept. Replaces the old 3..10 minimum, which was satisfied in v0.6 by attaching unrelated concepts. If the subject cannot be named, the object is not scoped." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:usesConcept ; sh:class pr:BusinessConcept ;
|
||||
sh:severity sh:Info ;
|
||||
sh:message "Other notions needed to read this object. Zero is a legitimate answer." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:monitoredBy ; sh:class pr:DataSteward ;
|
||||
sh:minCount 1 ;
|
||||
sh:severity sh:Warning ;
|
||||
sh:message "The Business Object is where a steward is declared -- Data Objects and Data Elements inherit from here." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:hasBusinessDefinition ; sh:maxCount 0 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "A Business Object carries no definition of its own: meaning belongs to the Business Concept named by aboutConcept. Two definitions for one notion are two definitions free to diverge. MIGRATION: move the text to that Concept." ] .
|
||||
|
||||
|
||||
#################################################################
|
||||
# BUSINESS CONCEPT AND METRIC
|
||||
#################################################################
|
||||
|
||||
prs:BusinessConceptShape a sh:NodeShape ;
|
||||
sh:targetClass pr:BusinessConcept ;
|
||||
rdfs:comment "Note what is NOT here: no requirement that a Concept be realized by any Data Element or measured by any Metric. The business layer is allowed to run ahead of the data. Coverage is a dashboard metric, not a rule." ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:hasBusinessDefinition ; sh:datatype xsd:string ;
|
||||
sh:minCount 1 ; sh:maxCount 1 ; sh:minLength 20 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "A Business Concept exists to carry meaning: it must hold exactly one business definition of at least 20 characters." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:hasFormula ; sh:maxCount 0 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "BR-003 -- the presence of a formula is what makes something a Metric. A Concept with a formula is a misclassified Metric." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:hasSource ; sh:maxCount 0 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "A Business Concept has no source of truth. Its authority is its owning domain and its definition belongs in the glossary. Naming a system here would ask which of the systems that USE the concept defines it -- a question with no answer." ] .
|
||||
|
||||
|
||||
prs:MetricShape a sh:NodeShape ;
|
||||
sh:targetClass pr:Metric ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:hasFormula ; sh:datatype xsd:string ;
|
||||
sh:minCount 1 ; sh:maxCount 1 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "BR-003 -- a Metric is a measurement with a calculation rule. No formula, no Metric." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:hasUnit ; sh:minCount 1 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "A Metric states every unit its variants are expressed in. NOT capped at one: Sell Out Volume is computed in 9L and in L, Sell Out Value in EUR, USD and local currency. The single-unit rule belongs on the Data Element, where it holds -- see DataElementShape." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:measures ; sh:class pr:BusinessConcept ;
|
||||
sh:minCount 1 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "A Metric measures at least one Business Concept -- this is how a measure element reaches business meaning." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:hasGranularity ; sh:maxCount 0 ;
|
||||
sh:severity sh:Warning ;
|
||||
sh:message "Grain moved off the Metric in v1.1. A Metric is a calculation rule and has no rows; grain belongs to the Data Object, stated with hasGrainElement and checkable against the real primary key." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:physicalName ; sh:maxCount 0 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "A Metric must not carry a physical name -- that is a layer leak. Under BR-013 the physical name belongs on the Data Element." ] .
|
||||
|
||||
|
||||
# The source of truth of a Metric is asserted by exception, then checked.
|
||||
# "Where can I get this metric" is a first-order question and the derivation
|
||||
# chain will have gaps before harvesting, so the assertion stands on its own --
|
||||
# but it must agree with the elements that implement the metric, or the two
|
||||
# will drift the day an element moves.
|
||||
prs:MetricSourceShape a sh:NodeShape ;
|
||||
sh:targetClass pr:Metric ;
|
||||
sh:sparql [
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "A Metric's source of truth must match the source of at least one element that computes it. If they disagree, either the metric points at the wrong system or an element has moved." ;
|
||||
sh:select """
|
||||
PREFIX pr: <https://ontology.pernod-ricard.com/metamodel/0.9/>
|
||||
PREFIX pr: <https://ontology.pernod-ricard.com/metamodel/>
|
||||
SELECT $this WHERE {
|
||||
$this pr:hasStatus "PUBLISHED" .
|
||||
$this pr:arbitrationStatus "TO_ARBITRATE" . }
|
||||
""" ] .
|
||||
$this pr:hasSource ?src ; pr:computedBy ?de .
|
||||
FILTER NOT EXISTS {
|
||||
$this pr:computedBy ?any .
|
||||
?any pr:hasSource ?src .
|
||||
}
|
||||
}""" ] ;
|
||||
sh:sparql [
|
||||
sh:severity sh:Warning ;
|
||||
sh:message "Elements computing one Metric come from several systems of truth. Legitimate for a metric blending sources, worth a look otherwise." ;
|
||||
sh:select """
|
||||
PREFIX pr: <https://ontology.pernod-ricard.com/metamodel/>
|
||||
SELECT $this WHERE {
|
||||
$this pr:computedBy ?a , ?b .
|
||||
?a pr:hasSource ?sa .
|
||||
?b pr:hasSource ?sb .
|
||||
FILTER(?sa != ?sb)
|
||||
}""" ] .
|
||||
|
||||
|
||||
# BR-013, finally executable. v0.1.1 declared it BLOCKING with no shape.
|
||||
prs:MotherMetricShape a sh:NodeShape ;
|
||||
sh:targetClass pr:Metric ;
|
||||
sh:sparql [
|
||||
sh:severity sh:Warning ;
|
||||
sh:message "BR-013 -- a PUBLISHED Metric should be implemented by at least one Data Element through computedBy. A metric no element computes is a definition with nothing behind it." ;
|
||||
sh:select """
|
||||
PREFIX pr: <https://ontology.pernod-ricard.com/metamodel/>
|
||||
SELECT $this WHERE {
|
||||
$this pr:hasStatus "PUBLISHED" .
|
||||
FILTER NOT EXISTS { $this pr:computedBy ?de }
|
||||
}""" ] .
|
||||
|
||||
|
||||
#################################################################
|
||||
# DATA OBJECT -- the pivot, and grain
|
||||
#################################################################
|
||||
|
||||
prs:DataObjectShape a sh:NodeShape ;
|
||||
sh:targetClass pr:DataObject ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:represents ; sh:class pr:BusinessObject ;
|
||||
sh:minCount 1 ; sh:maxCount 1 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "BR-006 -- a Data Object represents exactly one Business Object. This is the pivot: the single junction between governance and technology." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:hasElement ; sh:class pr:DataElement ;
|
||||
sh:minCount 1 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "A Data Object holds at least one Data Element." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:hasGrainElement ; sh:class pr:DataElement ;
|
||||
sh:severity sh:Warning ;
|
||||
sh:message "State the grain: the elements whose combination makes a row unique. Checkable against the harvested primary key." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:monitoredBy ; sh:maxCount 0 ;
|
||||
sh:severity sh:Warning ;
|
||||
sh:message "Stewardship is declared on the Business Object and inherited through represents. Declaring it here allows two answers to the same question." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:hasSource ; sh:maxCount 1 ;
|
||||
sh:not [ sh:pattern "/" ] ;
|
||||
sh:severity sh:Warning ;
|
||||
sh:message "hasSource names the SYSTEM of origin, nothing else. A value like 'SODH Gold / F_SELL_OUT' packs a system and a physical relation into one string -- the physical relation belongs in physicalName." ] .
|
||||
|
||||
|
||||
# A declared grain element must actually belong to the object it grains.
|
||||
prs:GrainConsistencyShape a sh:NodeShape ;
|
||||
sh:targetClass pr:DataObject ;
|
||||
sh:sparql [
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "A grain element must be one of the object's own Data Elements." ;
|
||||
sh:select """
|
||||
PREFIX pr: <https://ontology.pernod-ricard.com/metamodel/>
|
||||
SELECT $this ?grain WHERE {
|
||||
$this pr:hasGrainElement ?grain .
|
||||
FILTER NOT EXISTS { $this pr:hasElement ?grain }
|
||||
}""" ] .
|
||||
|
||||
|
||||
#################################################################
|
||||
# DATA ELEMENT -- the XOR of meaning
|
||||
#################################################################
|
||||
|
||||
prs:DataElementShape a sh:NodeShape ;
|
||||
sh:targetClass pr:DataElement ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:physicalName ; sh:datatype xsd:string ;
|
||||
sh:maxCount 1 ;
|
||||
sh:severity sh:Warning ;
|
||||
sh:message "NR-009 -- the physical name is recorded verbatim and never altered." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:fullyQualifiedName ; sh:maxCount 1 ;
|
||||
sh:severity sh:Info ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:hasUnit ; sh:maxCount 1 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "A Data Element carries exactly one unit. This is the counterpart of the Metric rule: the mother metric lists the units its variants use, each variant states the single unit it is actually expressed in. An element with two units is two elements." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:represents ; sh:class pr:BusinessConcept ;
|
||||
sh:maxCount 1 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "A Data Element represents at most one Business Concept. If a column carries two business notions, it should almost certainly be split -- so the case where a steward cannot choose is exactly the one worth looking at." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:monitoredBy ; sh:maxCount 0 ;
|
||||
sh:severity sh:Warning ;
|
||||
sh:message "Stewardship is inherited from the Business Object, not declared here." ] .
|
||||
|
||||
|
||||
# The core rule of v1.1.
|
||||
prs:DataElementMeaningShape a sh:NodeShape ;
|
||||
sh:targetClass pr:DataElement ;
|
||||
rdfs:comment "A Data Element reaches business meaning by exactly one of two routes: it represents a Business Concept (dimensional), or a Metric is computedBy it (measure). sh:xone rejects zero routes and two alike. A measure element must NOT also represent a Concept: the chain Metric -> measures -> Concept already carries the meaning, and carries more, since it names which metric it travels through." ;
|
||||
sh:xone (
|
||||
[ sh:property [ sh:path pr:represents ;
|
||||
sh:minCount 1 ; sh:maxCount 1 ] ]
|
||||
[ sh:property [ sh:path [ sh:inversePath pr:computedBy ] ;
|
||||
sh:minCount 1 ] ]
|
||||
) ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "A Data Element must reach business meaning by exactly one route: represents a Business Concept (dimensional), or be computedBy a Metric (measure). Never both, never neither. A curated column with no business meaning is debt, not an acceptable state." .
|
||||
|
||||
|
||||
#################################################################
|
||||
# DELIVERY
|
||||
#################################################################
|
||||
|
||||
prs:DataProductShape a sh:NodeShape ;
|
||||
sh:targetClass pr:DataProduct ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:operatedBy ; sh:class pr:SubDomain ;
|
||||
sh:minCount 1 ; sh:maxCount 1 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "A Data Product is operated by exactly one Sub-Domain. It spans several Business Objects and cannot inherit accountability, so it declares its own -- it is a consumable, and there must be no ambiguity about who runs it." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:hasProductOwner ; sh:class pr:DataProductOwner ;
|
||||
sh:maxCount 1 ;
|
||||
sh:severity sh:Warning ;
|
||||
sh:message "A Data Product should name exactly one Product Owner. A warning rather than a block: the SODH owner is genuinely not appointed, and a blocking rule that cannot be satisfied truthfully is how BR-004 produced padding." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:packages ; sh:class pr:DataObject ;
|
||||
sh:minCount 1 ;
|
||||
sh:severity sh:Violation ;
|
||||
sh:message "A Data Product packages at least one Data Object. The Business Objects it makes available are derived from these, which is why no direct link to a Business Object is needed." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:governedBy ; sh:maxCount 1 ;
|
||||
sh:severity sh:Warning ;
|
||||
sh:message "A Data Product should be governed by a Data Contract." ] ;
|
||||
|
||||
sh:property [
|
||||
sh:path pr:exposes ; sh:class pr:DataInterface ;
|
||||
sh:severity sh:Info ;
|
||||
sh:message "The channels through which consumers actually get the data." ] .
|
||||
|
||||
|
||||
prs:DataInterfaceShape a sh:NodeShape ;
|
||||
sh:targetClass pr:DataInterface ;
|
||||
sh:property [
|
||||
sh:path pr:servedBy ; sh:minCount 1 ;
|
||||
sh:severity sh:Warning ;
|
||||
sh:message "An interface should name what physically backs it. Unlike the old Data Asset test, this is verifiable: a grant, a share or an endpoint either exists or it does not." ] .
|
||||
|
||||
|
||||
#################################################################
|
||||
# CONSUMPTION
|
||||
#################################################################
|
||||
|
||||
prs:BIFieldShape a sh:NodeShape ;
|
||||
sh:targetClass pr:BIField ;
|
||||
rdfs:comment "realizesMetric is a transitional control: asserted directly, with no rule written against the lineage chain. Making the assertion conditional on intermediate mappings existing would create confusion and discourage people from filling it in at all." ;
|
||||
sh:property [
|
||||
sh:path pr:realizesMetric ; sh:class pr:Metric ;
|
||||
sh:maxCount 1 ;
|
||||
sh:severity sh:Info ;
|
||||
sh:message "Which certified Metric this BI field implements. Comparing expressions across fields realizing one Metric is the harmonisation measure." ] .
|
||||
|
||||
|
||||
#################################################################
|
||||
# NAMING -- portable regex, no lookbehind
|
||||
#################################################################
|
||||
|
||||
prs:NamingShape a sh:NodeShape ;
|
||||
sh:targetClass pr:CuratedObject ;
|
||||
sh:property [
|
||||
sh:path pr:hasName ;
|
||||
sh:not [ sh:pattern "_NM$" ] ;
|
||||
sh:severity sh:Warning ;
|
||||
sh:message "NR-001 -- a canonical name must not end in a physical suffix. Written as sh:not[sh:pattern] rather than a lookbehind: sh:pattern is normatively XPath 2.0 regex, which has no lookbehind, so the v0.1.1 form only worked under pySHACL." ] ;
|
||||
sh:property [
|
||||
sh:path pr:hasName ;
|
||||
sh:not [ sh:pattern "^[a-z_]+$" ] ;
|
||||
sh:severity sh:Warning ;
|
||||
sh:message "NR-002 -- a canonical name is written for humans, not as a snake_case identifier." ] .
|
||||
|
||||
Reference in New Issue
Block a user