tbox: v1.2 - physicalizedIn devient storedIn, hasSource ramene a un seul sens (systeme detenant la donnee faisant autorite) et interdit sur les Business Concepts. Table supprime, MaterializedView / constrainsRelation / approximateRowCount / hasVendor deprecies. shapes: controle que la source d'une metrique correspond a celle de ses elements, unite multiple sur Metric et unique sur DataElement, hasProductOwner en avertissement. instances: Data Product SODH avec contrat et deux interfaces, 6 proprietaires de domaine nommes, Sell Out Distribution devient Retail Distribution. generated: les deux viewers partagent desormais le meme moteur

This commit is contained in:
Bastien Gourdon
2026-07-29 09:03:59 +02:00
parent 7793def268
commit 76e95de471
9 changed files with 5314 additions and 1626 deletions
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+263 -148
View File
File diff suppressed because it is too large Load Diff
+66 -21
View File
@@ -7,11 +7,45 @@
@prefix pr: <https://ontology.pernod-ricard.com/metamodel/> . @prefix pr: <https://ontology.pernod-ricard.com/metamodel/> .
################################################################# #################################################################
# PERNOD RICARD DATA META MODEL - T-Box v1.1 # PERNOD RICARD DATA META MODEL - T-Box v1.2
# Owner: Bastien Gourdon - Data Governance Office # Owner: Bastien Gourdon - Data Governance Office
# #
# WHAT CHANGED vs v1.0 # WHAT CHANGED vs v1.1
# -------------------- # --------------------
# A. physicalizedIn -> storedIn. "Physicalized in" is jargon; the relation
# says where the data sits, and "stored in" says that to anyone.
#
# B. hasSource MEANS ONE THING: the system that holds the authoritative data.
# It was drifting into a third meaning on Business Concepts and Metrics,
# where it named the DOCUMENT a record came from ("SODH back-doc v0.7").
# A business concept has no data source -- it has an owning domain, which
# is its authority, and a definition published in the glossary. The
# property is now forbidden on concepts by SHACL.
#
# C. FIVE TERMS REMOVED OR DEPRECATED, none of them instantiated anywhere:
# Table deleted -- deprecated since v1.0, replaced by
# BaseTable, and never used. A deprecation that
# never ends stops meaning anything.
# constrainsRelation derivable: constrainsField -> livesIn gives the
# relation. An asserted edge redundant with an
# existing path is a future divergence.
# MaterializedView a class standing in for an attribute; being
# derived is what dependsOn already says.
# approximateRowCount volumetry with no identified use. queryCount30d
# and lastQueriedOn carry the rationalisation case.
# hasVendor marginal.
#
# D. hasProductOwner drops from BLOCKING to a warning in the shapes. The SODH
# product owner is genuinely not appointed yet, and a blocking rule that
# cannot be satisfied truthfully forces invention -- the BR-004 lesson.
#
# Environment and deployedIn stay, unused. Without them a DEV table and its
# PROD twin are indistinguishable and lineage pollutes silently on the first
# multi-environment harvest. Unused because nothing is harvested yet, not
# because they are useless.
#
# WHAT CHANGED vs v1.0 (kept for reference)
# -----------------------------------------
# A. RELATIONS SIMPLIFIED AND GIVEN HONEST NAMES # A. RELATIONS SIMPLIFIED AND GIVEN HONEST NAMES
# belongsToDomain + belongsToSubDomain -> belongsTo (one property, # belongsToDomain + belongsToSubDomain -> belongsTo (one property,
# scoped per class by SHACL; reusable if a domain family appears). # scoped per class by SHACL; reusable if a domain family appears).
@@ -117,12 +151,12 @@
<https://ontology.pernod-ricard.com/metamodel/> <https://ontology.pernod-ricard.com/metamodel/>
a owl:Ontology ; a owl:Ontology ;
owl:versionIRI <https://ontology.pernod-ricard.com/metamodel/1.1> ; owl:versionIRI <https://ontology.pernod-ricard.com/metamodel/1.2> ;
owl:priorVersion <https://ontology.pernod-ricard.com/metamodel/1.0> ; owl:priorVersion <https://ontology.pernod-ricard.com/metamodel/1.1> ;
dcterms:title "Pernod Ricard Data Meta Model" ; dcterms:title "Pernod Ricard Data Meta Model" ;
dcterms:creator "Bastien Gourdon" ; dcterms:creator "Bastien Gourdon" ;
dcterms:modified "2026-07-26"^^xsd:date ; dcterms:modified "2026-07-27"^^xsd:date ;
owl:versionInfo "1.1.0-draft-2026-07-26" ; owl:versionInfo "1.2.0-draft-2026-07-27" ;
rdfs:comment """OWL T-Box. Six layers (Ownership, Business, Logical, Delivery, Physical, Consumption) crossed with a provenance axis (Curated vs Observed). Curated objects are asserted by governance and reviewed; Observed objects are harvested from Snowflake, Tableau and Looker and are rebuilt, never edited. Bridge annotation properties are declared for future alignment with domain ontologies.""" . rdfs:comment """OWL T-Box. Six layers (Ownership, Business, Logical, Delivery, Physical, Consumption) crossed with a provenance axis (Curated vs Observed). Curated objects are asserted by governance and reviewed; Observed objects are harvested from Snowflake, Tableau and Looker and are rebuilt, never edited. Bridge annotation properties are declared for future alignment with domain ontologies.""" .
@@ -294,18 +328,14 @@ pr:View a owl:Class ; rdfs:subClassOf pr:PhysicalRelation ;
rdfs:comment "Depends on other relations through dependsOn, so impact is transitive." . rdfs:comment "Depends on other relations through dependsOn, so impact is transitive." .
pr:MaterializedView a owl:Class ; rdfs:subClassOf pr:PhysicalRelation ; pr:MaterializedView a owl:Class ; rdfs:subClassOf pr:PhysicalRelation ;
rdfs:label "Materialized View" . rdfs:label "Materialized View (deprecated)" ;
owl:deprecated true ; dcterms:isReplacedBy pr:View ;
rdfs:comment "A class standing in for an attribute: what distinguishes it is being derived, which dependsOn already states." .
pr:ExternalTable a owl:Class ; rdfs:subClassOf pr:PhysicalRelation ; pr:ExternalTable a owl:Class ; rdfs:subClassOf pr:PhysicalRelation ;
rdfs:label "External Table" ; rdfs:label "External Table" ;
rdfs:comment "Backed by files outside the warehouse. Marks the boundary where lineage leaves Snowflake." . rdfs:comment "Backed by files outside the warehouse. Marks the boundary where lineage leaves Snowflake." .
pr:Table a owl:Class ; rdfs:subClassOf pr:PhysicalRelation ;
rdfs:label "Table (deprecated)" ;
owl:deprecated true ;
dcterms:isReplacedBy pr:BaseTable ;
rdfs:comment "v0.9 term. Kept so that existing assertions still classify. Use BaseTable." .
pr:Field a owl:Class ; rdfs:subClassOf pr:PhysicalLayerObject ; pr:Field a owl:Class ; rdfs:subClassOf pr:PhysicalLayerObject ;
rdfs:label "Field" ; rdfs:label "Field" ;
pr:harvestSource "SNOWFLAKE.INFORMATION_SCHEMA.COLUMNS" ; pr:harvestSource "SNOWFLAKE.INFORMATION_SCHEMA.COLUMNS" ;
@@ -550,11 +580,15 @@ pr:materializedAs a owl:ObjectProperty ;
pr:curationMode "PROPOSED_CONFIRMED" ; pr:curationMode "PROPOSED_CONFIRMED" ;
rdfs:comment "Data Object to physical relation. Not functional: the same Data Object exists in DEV, UAT and PROD, and sometimes across several tables." . rdfs:comment "Data Object to physical relation. Not functional: the same Data Object exists in DEV, UAT and PROD, and sometimes across several tables." .
pr:physicalizedIn a owl:ObjectProperty ; pr:storedIn a owl:ObjectProperty ;
rdfs:label "physicalized in" ; rdfs:label "stored in" ;
rdfs:domain pr:DataElement ; rdfs:range pr:Field ; rdfs:domain pr:DataElement ; rdfs:range pr:Field ;
pr:curationMode "PROPOSED_CONFIRMED" ; pr:curationMode "PROPOSED_CONFIRMED" ;
rdfs:comment "Data Element to physical column, 1..n. Proposed by the reconciliation job on fullyQualifiedName and type, confirmed by a Data Steward. This single edge is what makes end-to-end impact analysis possible." . rdfs:comment "Data Element to physical column, 1..n. Proposed by the reconciliation job on fullyQualifiedName and type, confirmed by a Data Steward. This single edge is what makes end-to-end impact analysis possible. Renamed from physicalizedIn, which was jargon: the relation says where the data sits." .
pr:physicalizedIn a owl:ObjectProperty ;
rdfs:label "physicalized in (deprecated)" ;
owl:deprecated true ; dcterms:isReplacedBy pr:storedIn .
################################################################# #################################################################
@@ -597,8 +631,9 @@ pr:deployedIn a owl:ObjectProperty , owl:FunctionalProperty ;
rdfs:domain pr:ObservedObject ; rdfs:range pr:Environment . rdfs:domain pr:ObservedObject ; rdfs:range pr:Environment .
pr:constrainsRelation a owl:ObjectProperty , owl:FunctionalProperty ; pr:constrainsRelation a owl:ObjectProperty , owl:FunctionalProperty ;
rdfs:label "constrains relation" ; rdfs:label "constrains relation (deprecated)" ;
rdfs:domain pr:KeyConstraint ; rdfs:range pr:PhysicalRelation . owl:deprecated true ;
rdfs:comment "Derivable: constrainsField then livesIn gives the relation. An asserted edge redundant with an existing path is a divergence waiting to happen." .
pr:constrainsField a owl:ObjectProperty ; pr:constrainsField a owl:ObjectProperty ;
rdfs:label "constrains field" ; rdfs:label "constrains field" ;
@@ -738,7 +773,14 @@ pr:hasStatus a owl:DatatypeProperty ; rdfs:label "status" ; rdfs:range xsd:s
pr:hasVersion a owl:DatatypeProperty ; rdfs:label "version" ; rdfs:range xsd:string . pr:hasVersion a owl:DatatypeProperty ; rdfs:label "version" ; rdfs:range xsd:string .
pr:createdOn a owl:DatatypeProperty ; rdfs:label "created on" ; rdfs:range xsd:date . pr:createdOn a owl:DatatypeProperty ; rdfs:label "created on" ; rdfs:range xsd:date .
pr:lastReviewedOn a owl:DatatypeProperty ; rdfs:label "last reviewed on" ; rdfs:range xsd:date . pr:lastReviewedOn a owl:DatatypeProperty ; rdfs:label "last reviewed on" ; rdfs:range xsd:date .
pr:hasSource a owl:DatatypeProperty ; rdfs:label "back-doc source" ; rdfs:range xsd:string . pr:hasSource a owl:DatatypeProperty ; rdfs:label "source of truth" ; rdfs:range xsd:string ;
rdfs:comment """The system holding the authoritative data for this object: SODH Gold, MDM Product, Group Finance.
ONE MEANING ONLY. It had drifted into naming the document a record came from ("SODH back-doc v0.7"), and into packing a system and a relation into one string ("SODH Gold / F_SELL_OUT"). The relation belongs in physicalName; the document belongs nowhere -- provenance of a record is not a property of the thing it describes.
MUST NOT appear on a Business Concept: a concept has no data source. Its authority is its owning domain and its definition is published in the glossary. Enforced by SHACL.
On a Metric it is asserted rather than derived, by exception, because "where can I get this metric" is a first-order question and the derivation chain (computedBy, then the data object holding the element) will have gaps before harvesting. A shape checks the assertion against the elements.""" .
pr:arbitrationStatus a owl:DatatypeProperty ; rdfs:label "arbitration status" ; rdfs:range xsd:string ; pr:arbitrationStatus a owl:DatatypeProperty ; rdfs:label "arbitration status" ; rdfs:range xsd:string ;
rdfs:comment "TO_ARBITRATE blocks publication (OW-007). Use this on the object rather than tracking arbitration in a side spreadsheet." . rdfs:comment "TO_ARBITRATE blocks publication (OW-007). Use this on the object rather than tracking arbitration in a side spreadsheet." .
@@ -798,12 +840,15 @@ pr:hasPhysicalDataType a owl:DatatypeProperty ; rdfs:label "physical data type"
pr:isNullable a owl:DatatypeProperty ; rdfs:label "is nullable" ; rdfs:range xsd:boolean . pr:isNullable a owl:DatatypeProperty ; rdfs:label "is nullable" ; rdfs:range xsd:boolean .
pr:ordinalPosition a owl:DatatypeProperty ; rdfs:label "ordinal position" ; rdfs:range xsd:integer . pr:ordinalPosition a owl:DatatypeProperty ; rdfs:label "ordinal position" ; rdfs:range xsd:integer .
pr:hasViewDefinition a owl:DatatypeProperty ; rdfs:label "view definition" ; rdfs:range xsd:string . pr:hasViewDefinition a owl:DatatypeProperty ; rdfs:label "view definition" ; rdfs:range xsd:string .
pr:hasVendor a owl:DatatypeProperty ; rdfs:label "vendor" ; rdfs:range xsd:string . pr:hasVendor a owl:DatatypeProperty ; rdfs:label "vendor (deprecated)" ; rdfs:range xsd:string ;
owl:deprecated true .
pr:definedBy a owl:DatatypeProperty ; rdfs:label "defined by (code URI)" ; rdfs:range xsd:anyURI ; pr:definedBy a owl:DatatypeProperty ; rdfs:label "defined by (code URI)" ; rdfs:range xsd:anyURI ;
rdfs:comment "Git URI of the code behind a Transformation. Connects the graph to the repository." . rdfs:comment "Git URI of the code behind a Transformation. Connects the graph to the repository." .
# ---- Usage evidence (the rationalisation argument) # ---- Usage evidence (the rationalisation argument)
pr:approximateRowCount a owl:DatatypeProperty ; rdfs:label "approximate row count" ; rdfs:range xsd:integer . pr:approximateRowCount a owl:DatatypeProperty ; rdfs:label "approximate row count (deprecated)" ; rdfs:range xsd:integer ;
owl:deprecated true ;
rdfs:comment "Volumetry with no identified use. queryCount30d and lastQueriedOn carry the rationalisation argument; row count does not." .
pr:lastQueriedOn a owl:DatatypeProperty ; rdfs:label "last queried on" ; rdfs:range xsd:dateTime ; pr:lastQueriedOn a owl:DatatypeProperty ; rdfs:label "last queried on" ; rdfs:range xsd:dateTime ;
pr:harvestSource "SNOWFLAKE.ACCOUNT_USAGE.ACCESS_HISTORY" ; pr:harvestSource "SNOWFLAKE.ACCOUNT_USAGE.ACCESS_HISTORY" ;
rdfs:comment "Lets the model say 'documented but unused for eight months'. A rationalisation argument that costs nothing and is denominated in warehouse spend." . rdfs:comment "Lets the model say 'documented but unused for eight months'. A rationalisation argument that costs nothing and is denominated in warehouse spend." .
+202
View File
@@ -0,0 +1,202 @@
#!/usr/bin/env python3
"""
SODH - data fixes v1.2
=======================
Four corrections found while reviewing the viewer.
USAGE
python3 scripts/apply_fixes_v1_2.py # dry run
python3 scripts/apply_fixes_v1_2.py --apply # rewrite, .bak kept
1. DATA DOMAIN OWNERS
Six named owners added, plus the hasDomainOwner link. Ownership was
readable only through the DGL until now, which conflates two roles: the
owner is accountable for the domain, the DGL runs its governance.
2. hasSource CARRIED TWO MEANINGS
On a Data Element it named the system of origin (MDM Product, Group
Finance). On a Data Object it named the physical location (SODH Gold /
F_SELL_OUT). Same property, two concepts -- the defect we removed from
composedOf, still present one layer down. Split: physicalName holds the
relation name, hasSource holds the system. This is also what makes the
physical layer derivable: F_SELL_OUT plus 132 column names.
3. METRIC UNITS ARE PLURAL
Five mother metrics compute elements in several units (9L and L, EUR and
LC and USD). A metric states every unit its variants are expressed in; a
Data Element states exactly one. Recomputed from the elements rather than
asserted, so the two can never drift.
4. Sell Out Distribution -> Retail Distribution, matching the concept: sell
out counts units sold, distribution measures presence in store.
"""
import os
import re
import shutil
import sys
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
TTL = os.path.join(REPO, "instances", "sodh.ttl")
W = 78
OWNERS = [
("ex:DDO_DD_06", "Ludovic Cottier", "ex:DD_06"),
("ex:DDO_DD_04", "David Carpio", "ex:DD_04"),
("ex:DDO_DD_10", "Alberto Lupano", "ex:DD_10"),
("ex:DDO_DD_05", "David Carpio", "ex:DD_05"),
("ex:DDO_DD_16", "Julien Soisson", "ex:DD_16"),
("ex:DDO_DD_21", "Vincent Meunier", "ex:DD_21"),
]
RENAME = {"ex:BO_06_01_003": ("Sell Out Distribution", "Retail Distribution")}
# system of origin, once the physical location is taken out of it
SOURCE_FIX = {
"Snowflake (SODH Gold)": "SODH Gold",
"SODH Gold (flag)": "SODH Gold",
}
def clauses(body):
out, buf, i, n = [], [], 0, len(body)
while i < n:
c = body[i]
if c == '"':
j = i + 1
while j < n and body[j] != '"':
j += 2 if body[j] == "\\" else 1
j = min(j + 1, n)
buf.append(body[i:j]); i = j; continue
if c == ";":
out.append("".join(buf)); buf = []; i += 1; continue
buf.append(c); i += 1
out.append("".join(buf))
return out
def set_prop(text, subject, prop, value):
"""Clause-level set: survives mid-line properties and semicolons in literals."""
pat = re.compile(r'(^%s a pr:\w+ ;)(.*?)(\.\s*\n)' % re.escape(subject), re.S | re.M)
m = pat.search(text)
if not m:
return text, False
kept = [c for c in clauses(m.group(2))
if c.strip() and not re.match(r'\s*%s\s' % re.escape(prop), c)]
if value is not None:
kept.insert(0, "\n %s %s " % (prop, value))
body = " ;".join(kept).rstrip() + "\n "
return text[:m.start()] + m.group(1) + body + m.group(3) + text[m.end():], True
def get_prop(text, subject, prop):
m = re.search(r'^%s a pr:\w+ ;(.*?)\.\s*\n' % re.escape(subject), text, re.S | re.M)
if not m:
return None
for c in clauses(m.group(1)):
mm = re.match(r'\s*%s\s+(.*)$' % re.escape(prop), c, re.S)
if mm:
return mm.group(1).strip()
return None
def main():
apply_changes = "--apply" in sys.argv
if not os.path.exists(TTL):
print("Not found: %s" % TTL); sys.exit(2)
text = original = open(TTL, encoding="utf-8").read()
report = []
# ---- 1. domain owners -------------------------------------------------
todo = [o for o in OWNERS if not re.search(r'^%s a ' % re.escape(o[0]), text, re.M)]
if todo:
blocks = "\n".join('%s a pr:DataDomainOwner ; pr:hasName "%s" .' % (i, n)
for i, n, _ in todo)
anchor = re.search(r'^ex:ST_\w+ a pr:DataSteward', text, re.M)
pos = anchor.start() if anchor else 0
text = (text[:pos]
+ "# --- data domain owners ----------------------------------------------\n"
+ "# Accountable for the domain. Distinct from the DGL, who runs its\n"
+ "# governance: two roles that hasDGL alone was conflating.\n"
+ blocks + "\n\n" + text[pos:])
for i, n, dd in OWNERS:
text, _ = set_prop(text, dd, "pr:hasDomainOwner", i)
report.append("%d data domain owners named and linked" % len(OWNERS))
# ---- 2. physical location out of hasSource ----------------------------
n_do = 0
for m in list(re.finditer(r'^(ex:DO_\w+) a pr:DataObject ;', text, re.M)):
subj = m.group(1)
src = get_prop(text, subj, "pr:hasSource")
if src and "/" in src:
system, relation = [p.strip().strip('"') for p in src.strip('"').split("/", 1)]
text, _ = set_prop(text, subj, "pr:hasSource", '"%s"' % system)
text, _ = set_prop(text, subj, "pr:physicalName", '"%s"' % relation)
n_do += 1
report.append("%d data objects: physical relation moved out of hasSource" % n_do)
n_de = 0
for old, new in SOURCE_FIX.items():
text, k = re.subn(r'pr:hasSource "%s"' % re.escape(old),
'pr:hasSource "%s"' % new, text)
n_de += k
report.append("%d source labels normalised to the system of origin" % n_de)
# ---- 3. metric units, recomputed from the elements ---------------------
de_unit = {}
for m in re.finditer(r'^(ex:DE_\w+) a pr:DataElement ;(.*?)\.\s*\n', text, re.S | re.M):
u = re.search(r'pr:hasUnit\s+"([^"]*)"', m.group(2))
if u:
de_unit[m.group(1)] = u.group(1)
wired = {}
for m in re.finditer(r'(ex:M_\w+)\s+pr:computedBy(.*?)\.', text, re.S):
wired[m.group(1)] = re.findall(r'ex:DE_\w+', m.group(2))
n_m = 0
for metric, des in wired.items():
units = sorted({de_unit[d] for d in des if d in de_unit and de_unit[d] != "-"})
if not units:
continue
value = " , ".join('"%s"' % u for u in units)
before = get_prop(text, metric, "pr:hasUnit")
text, ok = set_prop(text, metric, "pr:hasUnit", value)
if ok and before != value:
n_m += 1
report.append("%d metrics now list every unit their elements use" % n_m)
# ---- 4. rename --------------------------------------------------------
for subj, (old, new) in RENAME.items():
if get_prop(text, subj, "pr:hasName") == '"%s"' % old:
text, _ = set_prop(text, subj, "pr:hasName", '"%s"' % new)
report.append("renamed %s -> %s" % (old, new))
# ---- report -----------------------------------------------------------
print()
print("SODH DATA FIXES v1.2 %s" % ("APPLY" if apply_changes else "DRY RUN"))
print("=" * W)
for line in report:
print(" " + line)
print("=" * W)
multi = len(re.findall(r'pr:hasUnit\s+"[^"]*"\s*,', text))
phys = len(re.findall(r'^ex:DO_\w+ a pr:DataObject ;(?:(?!\.\s*\n).)*?physicalName',
text, re.S | re.M))
doubled = []
for m in re.finditer(r'^(ex:\w+) a pr:\w+ ;(.*?)\.\s*\n', text, re.S | re.M):
for p in ("pr:hasName", "pr:hasSource", "pr:physicalName", "pr:hasDomainOwner"):
if len(re.findall(r'(?<![\w:])%s\s' % re.escape(p), m.group(2))) > 1:
doubled.append("%s/%s" % (m.group(1), p))
print(" metrics with several units %d | data objects with a physical name %d"
% (multi, phys))
print(" duplicated single-valued properties: %s"
% (", ".join(doubled) if doubled else "none"))
print("=" * W)
if text != original and apply_changes:
shutil.copy2(TTL, TTL + ".bak")
open(TTL, "w", encoding="utf-8").write(text)
print(" written, backup at %s.bak" % os.path.basename(TTL))
elif text != original:
print(" dry run -- re-run with --apply to write")
print()
if __name__ == "__main__":
main()
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+207
View File
@@ -0,0 +1,207 @@
#!/usr/bin/env python3
"""
SODH - migration v1.1 -> v1.2
==============================
USAGE
python3 scripts/migrate_instances_v1_2.py # dry run
python3 scripts/migrate_instances_v1_2.py --apply # rewrite, .bak kept
WHAT IT DOES
1. physicalizedIn -> storedIn everywhere (no instances yet, done for
completeness so the term never appears again).
2. hasSource removed from every Business Concept. A concept has no source
of truth: its authority is its owning domain, its definition belongs in
the glossary. Asking which system defines it has no answer, because
every system that uses it would be a candidate.
3. hasSource on Metrics recomputed from the elements that implement them.
It said "SODH back-doc v0.7" -- the provenance of the record, not the
source of the data. Asserted rather than derived, by exception, because
"where can I get this metric" is a first-order question and the
derivation chain will have gaps until harvesting; a shape then checks
the assertion against the elements.
4. The SODH Data Product, its contract and its two interfaces.
WHY THE PRODUCT OWNER IS EMPTY
Nobody is appointed. The field stays absent and the shape reports a
warning rather than a violation. Inventing a name to satisfy a blocking
rule is exactly how BR-004 produced padding in v0.6.
"""
import os
import re
import shutil
import sys
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
TTL = os.path.join(REPO, "instances", "sodh.ttl")
W = 78
PRODUCT = "ex:DP_06_01_001"
CONTRACT = "ex:DC_06_01_001"
IFACE_WH = "ex:DI_06_01_001"
IFACE_FILE = "ex:DI_06_01_002"
def clauses(body):
out, buf, i, n = [], [], 0, len(body)
while i < n:
c = body[i]
if c == '"':
j = i + 1
while j < n and body[j] != '"':
j += 2 if body[j] == "\\" else 1
j = min(j + 1, n)
buf.append(body[i:j]); i = j; continue
if c == ";":
out.append("".join(buf)); buf = []; i += 1; continue
buf.append(c); i += 1
out.append("".join(buf))
return out
def set_prop(text, subject, prop, value):
pat = re.compile(r'(^%s a pr:\w+ ;)(.*?)(\.\s*\n)' % re.escape(subject), re.S | re.M)
m = pat.search(text)
if not m:
return text, False
kept = [c for c in clauses(m.group(2))
if c.strip() and not re.match(r'\s*%s\s' % re.escape(prop), c)]
if value is not None:
kept.insert(0, "\n %s %s " % (prop, value))
body = " ;".join(kept).rstrip() + "\n "
return text[:m.start()] + m.group(1) + body + m.group(3) + text[m.end():], True
def get_prop(text, subject, prop):
m = re.search(r'^%s a pr:\w+ ;(.*?)\.\s*\n' % re.escape(subject), text, re.S | re.M)
if not m:
return None
for c in clauses(m.group(1)):
mm = re.match(r'\s*%s\s+(.*)$' % re.escape(prop), c, re.S)
if mm:
return mm.group(1).strip()
return None
def product_blocks(objects):
packages = " ,\n ".join(objects)
return """# --- data product ------------------------------------------------------
# The unit of delivery: what carries a version, an owner, a changelog and a
# list of consumers to warn. It has no single technical counterpart on purpose
# -- rename the schema and the product is unchanged; change its SLA and it is
# not. Its physical footprint is derivable through packages, never asserted.
%s a pr:DataProduct ;
pr:hasIdentifier "DP-06.01-001" ; pr:hasName "Sell Out Data Hub - Gold" ;
pr:owningDomain ex:DD_06 ;
pr:operatedBy ex:SD_06_01 ;
pr:packages %s ;
pr:governedBy %s ;
pr:exposes %s , %s ;
pr:hasStatus "DRAFT" ; pr:hasVersion "1.2" ; pr:hasSource "SODH Gold" .
%s a pr:DataContract ;
pr:hasIdentifier "DC-06.01-001" ; pr:hasName "Sell Out Data Hub - Gold service commitment" ;
pr:owningDomain ex:DD_06 ;
pr:hasBusinessRule "Refreshed weekly, available Tuesday 06:00 CET. Panel coverage published with every load. Schema stable within a minor version; 60 days notice on any breaking change. Support through the Sales Performance data team." ;
pr:hasStatus "DRAFT" ; pr:hasVersion "1.2" .
%s a pr:DataInterface ;
pr:hasIdentifier "DI-06.01-001" ; pr:hasName "SODH Gold consumption schema" ;
pr:owningDomain ex:DD_06 ;
pr:hasBusinessDefinition "Read access to the Gold layer through Snowflake grants. The channel BI tools and analysts consume."@en ;
pr:hasStatus "DRAFT" ; pr:hasVersion "1.2" ; pr:hasSource "SODH Gold" .
%s a pr:DataInterface ;
pr:hasIdentifier "DI-06.01-002" ; pr:hasName "Panel partner weekly extract" ;
pr:owningDomain ex:DD_06 ;
pr:hasBusinessDefinition "File extract delivered to the panel provider each week. Outside Snowflake, and the boundary where lineage leaves the warehouse."@en ;
pr:hasStatus "DRAFT" ; pr:hasVersion "1.2" ; pr:hasSource "SODH Gold" .
""" % (PRODUCT, packages, CONTRACT, IFACE_WH, IFACE_FILE,
CONTRACT, IFACE_WH, IFACE_FILE)
def main():
apply_changes = "--apply" in sys.argv
if not os.path.exists(TTL):
print("Not found: %s" % TTL); sys.exit(2)
text = original = open(TTL, encoding="utf-8").read()
report = []
# 1. term rename
text, n = re.subn(r'\bpr:physicalizedIn\b', 'pr:storedIn', text)
report.append("physicalizedIn -> storedIn: %d occurrence(s)" % n)
# 2. no source of truth on a concept
n_bc = 0
for m in list(re.finditer(r'^(ex:BC_\w+) a pr:BusinessConcept ;', text, re.M)):
if get_prop(text, m.group(1), "pr:hasSource") is not None:
text, _ = set_prop(text, m.group(1), "pr:hasSource", None)
n_bc += 1
report.append("hasSource removed from %d business concepts" % n_bc)
# 3. metric source of truth, from the elements that implement it
de_src = {}
for m in re.finditer(r'^(ex:DE_\w+) a pr:DataElement ;(.*?)\.\s*\n', text, re.S | re.M):
v = re.search(r'pr:hasSource\s+"([^"]*)"', m.group(2))
if v:
de_src[m.group(1)] = v.group(1)
wired = {}
for m in re.finditer(r'(ex:M_\w+)\s+pr:computedBy(.*?)\.', text, re.S):
wired[m.group(1)] = re.findall(r'ex:DE_\w+', m.group(2))
n_m, mixed = 0, []
for metric, des in wired.items():
srcs = sorted({de_src[d] for d in des if d in de_src})
if not srcs:
continue
if len(srcs) > 1:
mixed.append(metric)
text, ok = set_prop(text, metric, "pr:hasSource",
" , ".join('"%s"' % x for x in srcs))
n_m += ok
report.append("source of truth recomputed on %d metrics%s"
% (n_m, "" if not mixed else
" (%d blend several systems)" % len(mixed)))
# 4. the data product
if not re.search(r'^%s a ' % re.escape(PRODUCT), text, re.M):
objs = re.findall(r'^(ex:DO_\w+) a pr:DataObject ;', text, re.M)
text = text.rstrip() + "\n\n\n" + product_blocks(objs)
report.append("data product created, packaging %d data objects" % len(objs))
else:
report.append("data product already present, skipped")
# ---- report
print()
print("SODH MIGRATION v1.1 -> v1.2 %s" % ("APPLY" if apply_changes else "DRY RUN"))
print("=" * W)
for line in report:
print(" " + line)
print("=" * W)
bc_src = len(re.findall(r'^ex:BC_\w+ a pr:BusinessConcept ;(?:(?!\.\s*\n).)*?hasSource',
text, re.S | re.M))
defined = set(re.findall(r'^(ex:\w+) a ', text, re.M))
dangling = sorted(set(re.findall(r'\bex:\w+', text)) - defined)
doubled = []
for m in re.finditer(r'^(ex:\w+) a pr:\w+ ;(.*?)\.\s*\n', text, re.S | re.M):
for p in ("pr:hasName", "pr:hasSource", "pr:operatedBy", "pr:governedBy"):
if len(re.findall(r'(?<![\w:])%s\s' % re.escape(p), m.group(2))) > 1:
doubled.append("%s/%s" % (m.group(1), p))
print(" concepts still carrying a source: %d (must be 0)" % bc_src)
print(" physicalizedIn left: %d (must be 0)"
% len(re.findall(r'pr:physicalizedIn', text)))
print(" dangling references: %s" % (", ".join(dangling) if dangling else "none"))
print(" duplicated single-valued properties: %s"
% (", ".join(doubled) if doubled else "none"))
print("=" * W)
if text != original and apply_changes:
shutil.copy2(TTL, TTL + ".bak")
open(TTL, "w", encoding="utf-8").write(text)
print(" written, backup at %s.bak" % os.path.basename(TTL))
elif text != original:
print(" dry run -- re-run with --apply to write")
print()
if __name__ == "__main__":
main()
+56 -8
View File
@@ -40,7 +40,7 @@
<https://ontology.pernod-ricard.com/metamodel/shapes/> <https://ontology.pernod-ricard.com/metamodel/shapes/>
dcterms:title "PR Data Meta Model - SHACL Shapes" ; dcterms:title "PR Data Meta Model - SHACL Shapes" ;
dcterms:modified "2026-07-26"^^xsd:date ; dcterms:modified "2026-07-27"^^xsd:date ;
rdfs:comment "Executable form of the rulebook. Validated by scripts/run_shacl_validation.py." . rdfs:comment "Executable form of the rulebook. Validated by scripts/run_shacl_validation.py." .
@@ -231,7 +231,12 @@ prs:BusinessConceptShape a sh:NodeShape ;
sh:property [ sh:property [
sh:path pr:hasFormula ; sh:maxCount 0 ; sh:path pr:hasFormula ; sh:maxCount 0 ;
sh:severity sh:Violation ; sh:severity sh:Violation ;
sh:message "BR-003 -- the presence of a formula is what makes something a Metric. A Concept with a formula is a misclassified Metric." ] . sh:message "BR-003 -- the presence of a formula is what makes something a Metric. A Concept with a formula is a misclassified Metric." ] ;
sh:property [
sh:path pr:hasSource ; sh:maxCount 0 ;
sh:severity sh:Violation ;
sh:message "A Business Concept has no source of truth. Its authority is its owning domain and its definition belongs in the glossary. Naming a system here would ask which of the systems that USE the concept defines it -- a question with no answer." ] .
prs:MetricShape a sh:NodeShape ; prs:MetricShape a sh:NodeShape ;
@@ -244,9 +249,9 @@ prs:MetricShape a sh:NodeShape ;
sh:message "BR-003 -- a Metric is a measurement with a calculation rule. No formula, no Metric." ] ; sh:message "BR-003 -- a Metric is a measurement with a calculation rule. No formula, no Metric." ] ;
sh:property [ sh:property [
sh:path pr:hasUnit ; sh:minCount 1 ; sh:maxCount 1 ; sh:path pr:hasUnit ; sh:minCount 1 ;
sh:severity sh:Violation ; sh:severity sh:Violation ;
sh:message "A Metric states its unit." ] ; sh:message "A Metric states every unit its variants are expressed in. NOT capped at one: Sell Out Volume is computed in 9L and in L, Sell Out Value in EUR, USD and local currency. The single-unit rule belongs on the Data Element, where it holds -- see DataElementShape." ] ;
sh:property [ sh:property [
sh:path pr:measures ; sh:class pr:BusinessConcept ; sh:path pr:measures ; sh:class pr:BusinessConcept ;
@@ -265,6 +270,38 @@ prs:MetricShape a sh:NodeShape ;
sh:message "A Metric must not carry a physical name -- that is a layer leak. Under BR-013 the physical name belongs on the Data Element." ] . sh:message "A Metric must not carry a physical name -- that is a layer leak. Under BR-013 the physical name belongs on the Data Element." ] .
# The source of truth of a Metric is asserted by exception, then checked.
# "Where can I get this metric" is a first-order question and the derivation
# chain will have gaps before harvesting, so the assertion stands on its own --
# but it must agree with the elements that implement the metric, or the two
# will drift the day an element moves.
prs:MetricSourceShape a sh:NodeShape ;
sh:targetClass pr:Metric ;
sh:sparql [
sh:severity sh:Violation ;
sh:message "A Metric's source of truth must match the source of at least one element that computes it. If they disagree, either the metric points at the wrong system or an element has moved." ;
sh:select """
PREFIX pr: <https://ontology.pernod-ricard.com/metamodel/>
SELECT $this WHERE {
$this pr:hasSource ?src ; pr:computedBy ?de .
FILTER NOT EXISTS {
$this pr:computedBy ?any .
?any pr:hasSource ?src .
}
}""" ] ;
sh:sparql [
sh:severity sh:Warning ;
sh:message "Elements computing one Metric come from several systems of truth. Legitimate for a metric blending sources, worth a look otherwise." ;
sh:select """
PREFIX pr: <https://ontology.pernod-ricard.com/metamodel/>
SELECT $this WHERE {
$this pr:computedBy ?a , ?b .
?a pr:hasSource ?sa .
?b pr:hasSource ?sb .
FILTER(?sa != ?sb)
}""" ] .
# BR-013, finally executable. v0.1.1 declared it BLOCKING with no shape. # BR-013, finally executable. v0.1.1 declared it BLOCKING with no shape.
prs:MotherMetricShape a sh:NodeShape ; prs:MotherMetricShape a sh:NodeShape ;
sh:targetClass pr:Metric ; sh:targetClass pr:Metric ;
@@ -306,7 +343,13 @@ prs:DataObjectShape a sh:NodeShape ;
sh:property [ sh:property [
sh:path pr:monitoredBy ; sh:maxCount 0 ; sh:path pr:monitoredBy ; sh:maxCount 0 ;
sh:severity sh:Warning ; sh:severity sh:Warning ;
sh:message "Stewardship is declared on the Business Object and inherited through represents. Declaring it here allows two answers to the same question." ] . sh:message "Stewardship is declared on the Business Object and inherited through represents. Declaring it here allows two answers to the same question." ] ;
sh:property [
sh:path pr:hasSource ; sh:maxCount 1 ;
sh:not [ sh:pattern "/" ] ;
sh:severity sh:Warning ;
sh:message "hasSource names the SYSTEM of origin, nothing else. A value like 'SODH Gold / F_SELL_OUT' packs a system and a physical relation into one string -- the physical relation belongs in physicalName." ] .
# A declared grain element must actually belong to the object it grains. # A declared grain element must actually belong to the object it grains.
@@ -340,6 +383,11 @@ prs:DataElementShape a sh:NodeShape ;
sh:path pr:fullyQualifiedName ; sh:maxCount 1 ; sh:path pr:fullyQualifiedName ; sh:maxCount 1 ;
sh:severity sh:Info ] ; sh:severity sh:Info ] ;
sh:property [
sh:path pr:hasUnit ; sh:maxCount 1 ;
sh:severity sh:Violation ;
sh:message "A Data Element carries exactly one unit. This is the counterpart of the Metric rule: the mother metric lists the units its variants use, each variant states the single unit it is actually expressed in. An element with two units is two elements." ] ;
sh:property [ sh:property [
sh:path pr:represents ; sh:class pr:BusinessConcept ; sh:path pr:represents ; sh:class pr:BusinessConcept ;
sh:maxCount 1 ; sh:maxCount 1 ;
@@ -381,9 +429,9 @@ prs:DataProductShape a sh:NodeShape ;
sh:property [ sh:property [
sh:path pr:hasProductOwner ; sh:class pr:DataProductOwner ; sh:path pr:hasProductOwner ; sh:class pr:DataProductOwner ;
sh:minCount 1 ; sh:maxCount 1 ; sh:maxCount 1 ;
sh:severity sh:Violation ; sh:severity sh:Warning ;
sh:message "A Data Product has exactly one Product Owner." ] ; sh:message "A Data Product should name exactly one Product Owner. A warning rather than a block: the SODH owner is genuinely not appointed, and a blocking rule that cannot be satisfied truthfully is how BR-004 produced padding." ] ;
sh:property [ sh:property [
sh:path pr:packages ; sh:class pr:DataObject ; sh:path pr:packages ; sh:class pr:DataObject ;