tbox: v1.1 - belongsTo unifie, implements devient represents, aboutConcept/usesConcept remplacent composedOf, monitoredBy sur BO seul, grain deplace du Metric au DataObject, couche Delivery et DataInterface. shapes v0.2 scindees curated/observed avec XOR de sens sur DataElement. instances SODH migrees et arbitrees : 12 BO, 12 BC, 6 concepts empruntes en TO_ARBITRATE. Validation : 52 bloquants, dont 52 leves par la migration BR-013 restante
This commit is contained in:
Bastien Gourdon
@@ -0,0 +1,255 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
SODH arbitration - composedOf -> aboutConcept + usesConcept
|
||||
============================================================
|
||||
Second half of the v1.0 -> v1.1 migration. migrate_instances_v1_1.py did the
|
||||
mechanical renames and deliberately stopped here, because everything below is a
|
||||
judgement someone had to make.
|
||||
|
||||
USAGE
|
||||
python3 scripts/apply_arbitration_sodh.py # dry run
|
||||
python3 scripts/apply_arbitration_sodh.py --apply # rewrite, .bak kept
|
||||
|
||||
WHAT WAS ARBITRATED (Bastien, 2026-07-27)
|
||||
-----------------------------------------
|
||||
The v0.6 padding is what made this necessary: four Business Objects belonging
|
||||
to other domains -- Product Structure, Customer & Outlet, Currency, Calendar --
|
||||
were all declared composedOf the same three Sell Out concepts, because BR-004
|
||||
demanded three and there were none to give. The subject rule replaces that
|
||||
minimum: exactly one subject, and if it cannot be named the object is not
|
||||
scoped.
|
||||
|
||||
1. Distribution -> Retail Distribution.
|
||||
"Distribution" is too generic to survive contact with other domains, and
|
||||
it is NOT a kind of sell out: sell out counts units sold, distribution
|
||||
measures product presence in store. Retail Distribution follows panel
|
||||
vocabulary (Nielsen/Circana), under which Numeric and Weighted
|
||||
Distribution are the metrics.
|
||||
|
||||
2. Sell Out Baseline becomes a concept in its own right, not a flavour of
|
||||
Sell Out: modelled non-promoted volume is a different notion from
|
||||
observed volume.
|
||||
|
||||
3. Customer & Outlet splits into two Business Objects, and Currency &
|
||||
Exchange Rates likewise. An object named "X & Y" has no single subject,
|
||||
which is precisely what the rule is meant to surface. The split is
|
||||
PROPOSED here; DD-04 and DD-16 own the decision.
|
||||
|
||||
4. Six concepts belonging to other domains are created with a DRAFT
|
||||
definition and arbitrationStatus TO_ARBITRATE. A draft to correct beats a
|
||||
blank to fill, and OW-007 blocks publication until the owning domain has
|
||||
ratified. These definitions are the DGO's proposal, not DD-04/10/16/21's
|
||||
word.
|
||||
|
||||
NOT DONE HERE
|
||||
hasGranularity on Metrics still needs re-expressing as hasGrainElement on
|
||||
Data Objects, and the BR-013 alignment (10 mother metrics, computedBy)
|
||||
is a separate migration. Until that lands, every Data Element will fail
|
||||
the XOR-of-meaning shape -- which is the correct, informative result.
|
||||
"""
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
TARGET = os.path.join(REPO, "instances", "sodh.ttl")
|
||||
W = 78
|
||||
|
||||
# ---------------------------------------------------------------- new concepts
|
||||
# (iri, identifier, name, domain, steward, definition, arbitrate)
|
||||
NEW_CONCEPTS = [
|
||||
("ex:BC_06_01_005", "BC-06.01-005", "Sell Out Baseline", "ex:DD_06", "ex:ST_DD_06",
|
||||
"The modelled level of retail sales that would have occurred without promotional activity, "
|
||||
"used as the reference against which promotional uplift is measured. Distinct from Sell Out, "
|
||||
"which is observed rather than modelled.",
|
||||
False),
|
||||
("ex:BC_10_01_001", "BC-10.01-001", "Product", "ex:DD_10", "ex:ST_DD_10",
|
||||
"A commercial item sold by Pernod Ricard, identified at Stock Keeping Unit level and "
|
||||
"positioned in the corporate hierarchy from Brand through Category, Sub-Category and Flavor.",
|
||||
True),
|
||||
("ex:BC_04_01_001", "BC-04.01-001", "Customer", "ex:DD_04", "ex:ST_DD_04",
|
||||
"A commercial counterparty that buys Pernod Ricard products for resale, positioned in the "
|
||||
"trade hierarchy from Distributor through Tier-1 and Tier-2 Customer. Distinct from Outlet, "
|
||||
"which is a physical point of sale.",
|
||||
True),
|
||||
("ex:BC_04_01_002", "BC-04.01-002", "Outlet", "ex:DD_04", "ex:ST_DD_04",
|
||||
"A physical point of sale where products are made available to end consumers, characterised "
|
||||
"by its Channel. An Outlet belongs to a Customer but is not the same notion: one Customer "
|
||||
"operates many Outlets.",
|
||||
True),
|
||||
("ex:BC_16_01_001", "BC-16.01-001", "Currency", "ex:DD_16", "ex:ST_DD_16",
|
||||
"A monetary unit in which a transaction or a reported amount is denominated, identified by "
|
||||
"its ISO 4217 code.",
|
||||
True),
|
||||
("ex:BC_16_01_002", "BC-16.01-002", "Exchange Rate", "ex:DD_16", "ex:ST_DD_16",
|
||||
"The conversion factor applied between two currencies at a given date, used to restate local "
|
||||
"amounts into group reporting currency. Distinct from Currency: a rate is a dated "
|
||||
"relationship between two of them.",
|
||||
True),
|
||||
("ex:BC_21_01_001", "BC-21.01-001", "Calendar Date", "ex:DD_21", "ex:ST_DD_21",
|
||||
"A single day in the Gregorian calendar, the atomic unit from which Week, Month, Quarter and "
|
||||
"Year are derived and against which all time-based measures are positioned.",
|
||||
True),
|
||||
]
|
||||
|
||||
# ------------------------------------------------------------------- new object
|
||||
NEW_OBJECTS = [
|
||||
("ex:BO_04_01_002", "BO-04.01-002", "Outlet", "ex:DD_04", "ex:ST_DD_04", "ex:SD_04_01",
|
||||
"ex:BC_04_01_002", [], "MDM Customer -> SODH [split from BO-04.01-001, proposed to DD-04]"),
|
||||
("ex:BO_16_01_002", "BO-16.01-002", "Exchange Rate", "ex:DD_16", "ex:ST_DD_16", "ex:SD_16_01",
|
||||
"ex:BC_16_01_002", [], "Group Finance -> SODH [split from BO-16.01-001, proposed to DD-16]"),
|
||||
]
|
||||
|
||||
# ---------------------------------------------------- subject / used per object
|
||||
# iri -> (aboutConcept, [usesConcept...], new name or None)
|
||||
ARBITRATION = {
|
||||
"ex:BO_06_01_001": ("ex:BC_06_01_001", ["ex:BC_06_01_003", "ex:BC_05_01_001"], None),
|
||||
"ex:BO_06_01_002": ("ex:BC_06_01_001", ["ex:BC_06_01_003", "ex:BC_05_01_001",
|
||||
"ex:BC_16_01_001"], None),
|
||||
"ex:BO_06_01_003": ("ex:BC_06_01_004", ["ex:BC_06_01_003"], None),
|
||||
"ex:BO_06_01_004": ("ex:BC_06_01_002", ["ex:BC_06_01_001", "ex:BC_06_01_003"], None),
|
||||
"ex:BO_06_01_005": ("ex:BC_06_01_005", ["ex:BC_06_01_001", "ex:BC_05_01_001"], None),
|
||||
"ex:BO_10_01_001": ("ex:BC_10_01_001", [], None),
|
||||
"ex:BO_04_01_001": ("ex:BC_04_01_001", [], "Customer"),
|
||||
"ex:BO_16_01_001": ("ex:BC_16_01_001", [], "Currency"),
|
||||
"ex:BO_21_01_001": ("ex:BC_21_01_001", [], None),
|
||||
"ex:BO_05_01_001": ("ex:BC_05_01_001", [], None),
|
||||
}
|
||||
|
||||
RENAME_CONCEPT = {"ex:BC_06_01_004": ("Distribution", "Retail Distribution")}
|
||||
|
||||
|
||||
def concept_block(iri, ident, name, domain, steward, definition, arbitrate):
|
||||
lines = [
|
||||
'%s a pr:BusinessConcept ;' % iri,
|
||||
' pr:hasIdentifier "%s" ; pr:hasName "%s" ;' % (ident, name),
|
||||
' pr:owningDomain %s ; pr:ownedBy %s ;' % (domain, steward),
|
||||
' pr:hasBusinessDefinition "%s" ;' % definition,
|
||||
]
|
||||
if arbitrate:
|
||||
lines.append(' pr:arbitrationStatus "TO_ARBITRATE" ;')
|
||||
src = "DGO proposal, pending ratification by the owning domain"
|
||||
else:
|
||||
src = "SODH Gold"
|
||||
lines.append(' pr:hasStatus "DRAFT" ; pr:hasVersion "1.1" ; pr:hasSource "%s" .' % src)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def object_block(iri, ident, name, domain, steward, subdomain, about, uses, source):
|
||||
lines = [
|
||||
'%s a pr:BusinessObject ;' % iri,
|
||||
' pr:hasIdentifier "%s" ; pr:hasName "%s" ;' % (ident, name),
|
||||
' pr:owningDomain %s ; pr:ownedBy %s ;' % (domain, steward),
|
||||
' pr:belongsTo %s ; pr:monitoredBy %s ;' % (subdomain, steward),
|
||||
' pr:aboutConcept %s ;' % about,
|
||||
]
|
||||
if uses:
|
||||
lines.append(' pr:usesConcept %s ;' % " , ".join(uses))
|
||||
lines.append(' pr:arbitrationStatus "TO_ARBITRATE" ;')
|
||||
lines.append(' pr:hasStatus "DRAFT" ; pr:hasVersion "1.1" ; pr:hasSource "%s" .' % source)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main():
|
||||
apply_changes = "--apply" in sys.argv
|
||||
if not os.path.exists(TARGET):
|
||||
print("Not found: %s" % TARGET)
|
||||
sys.exit(2)
|
||||
|
||||
text = original = open(TARGET, encoding="utf-8").read()
|
||||
report = []
|
||||
|
||||
# ---- 1. rename the over-generic concept ----------------------------------
|
||||
for iri, (old, new) in RENAME_CONCEPT.items():
|
||||
pat = re.compile(r'(%s\b.*?pr:hasName )"%s"' % (re.escape(iri), re.escape(old)), re.S)
|
||||
text, n = pat.subn(r'\1"%s"' % new, text, count=1)
|
||||
if n:
|
||||
report.append("renamed concept %s: %s -> %s" % (iri, old, new))
|
||||
|
||||
# ---- 2. rewrite each Business Object -------------------------------------
|
||||
for iri, (about, uses, newname) in ARBITRATION.items():
|
||||
m = re.search(r'(^%s a pr:BusinessObject ;.*?\n)(?=^ex:|\Z)' % re.escape(iri),
|
||||
text, re.S | re.M)
|
||||
if not m:
|
||||
report.append("!! block not found: %s" % iri)
|
||||
continue
|
||||
block = m.group(1)
|
||||
|
||||
new_block = re.sub(r'[ \t]*pr:composedOf\s+[^";]*;[ \t]*\n', '', block)
|
||||
clause = ' pr:aboutConcept %s ;\n' % about
|
||||
if uses:
|
||||
clause += ' pr:usesConcept %s ;\n' % " , ".join(uses)
|
||||
new_block = re.sub(r'(\n)([ \t]*pr:hasMetric|[ \t]*pr:hasBusinessDefinition|[ \t]*pr:hasStatus)',
|
||||
r'\1%s\2' % clause, new_block, count=1)
|
||||
|
||||
# meaning belongs to the Concept, never to the Object
|
||||
# match the full quoted literal: several definitions contain a semicolon
|
||||
# inside the string ("Owned by Product & Material; consumed by SODH."),
|
||||
# which a naive [^;]* would stop at, silently leaving the property behind
|
||||
new_block, ndef = re.subn(
|
||||
r'[ \t]*pr:hasBusinessDefinition\s+"(?:[^"\\]|\\.)*"\s*;[ \t]*\n',
|
||||
'', new_block)
|
||||
|
||||
if newname:
|
||||
new_block = re.sub(r'(pr:hasName )"[^"]*"', r'\1"%s"' % newname, new_block, count=1)
|
||||
new_block = re.sub(r'(pr:hasSource )"[^"]*"',
|
||||
r'\1"MDM -> SODH [split proposed, subject arbitrated 2026-07-27]"',
|
||||
new_block, count=1)
|
||||
|
||||
text = text[:m.start(1)] + new_block + text[m.end(1):]
|
||||
report.append("%-18s about=%s uses=%d%s%s"
|
||||
% (iri, about.replace("ex:BC_", "BC-"), len(uses),
|
||||
" def->concept" if ndef else "",
|
||||
" renamed=%s" % newname if newname else ""))
|
||||
|
||||
# ---- 3. insert the missing concepts --------------------------------------
|
||||
anchor = re.search(r'\n(?=ex:BO_\w+ a pr:BusinessObject ;)', text)
|
||||
if anchor:
|
||||
blocks = "\n".join(concept_block(*c) for c in NEW_CONCEPTS)
|
||||
text = (text[:anchor.start()] + "\n\n" +
|
||||
"# --- concepts added in v1.1 migration -------------------------------\n" +
|
||||
"# Six of these belong to other domains: DRAFT definitions proposed by the\n" +
|
||||
"# DGO, flagged TO_ARBITRATE. OW-007 blocks publication until ratified.\n" +
|
||||
blocks + "\n" + text[anchor.start():])
|
||||
report.append("inserted %d Business Concepts (%d TO_ARBITRATE)"
|
||||
% (len(NEW_CONCEPTS), sum(1 for c in NEW_CONCEPTS if c[6])))
|
||||
|
||||
# ---- 4. append the objects produced by the splits ------------------------
|
||||
tail = re.search(r'\n(?=ex:M_\w+ a pr:Metric ;|ex:DO_\w+ a pr:DataObject ;)', text)
|
||||
if tail:
|
||||
blocks = "\n".join(object_block(*o) for o in NEW_OBJECTS)
|
||||
text = (text[:tail.start()] + "\n\n" +
|
||||
"# --- Business Objects from the v1.1 splits ---------------------------\n" +
|
||||
"# 'Customer & Outlet' and 'Currency & Exchange Rates' had no single\n" +
|
||||
"# subject. Split proposed to DD-04 and DD-16, hence TO_ARBITRATE.\n" +
|
||||
blocks + "\n" + text[tail.start():])
|
||||
report.append("inserted %d Business Objects from splits" % len(NEW_OBJECTS))
|
||||
|
||||
# ---- report --------------------------------------------------------------
|
||||
print()
|
||||
print("SODH ARBITRATION %s" % ("APPLY" if apply_changes else "DRY RUN"))
|
||||
print("=" * W)
|
||||
for line in report:
|
||||
print(" " + line)
|
||||
print("=" * W)
|
||||
bo = len(re.findall(r'a pr:BusinessObject', text))
|
||||
bc = len(re.findall(r'a pr:BusinessConcept', text))
|
||||
print(" Business Objects %d | Business Concepts %d | composedOf left %d"
|
||||
% (bo, bc, len(re.findall(r'pr:composedOf', text))))
|
||||
print(" Business Objects still carrying a definition: %d"
|
||||
% len([b for b in re.split(r'\.\s*\n(?=\S)', text)
|
||||
if "pr:BusinessObject" in b and "pr:hasBusinessDefinition" in b]))
|
||||
print("=" * W)
|
||||
|
||||
if text != original and apply_changes:
|
||||
shutil.copy2(TARGET, TARGET + ".bak")
|
||||
open(TARGET, "w", encoding="utf-8").write(text)
|
||||
print(" written, backup at %s.bak" % os.path.basename(TARGET))
|
||||
elif text != original:
|
||||
print(" dry run -- re-run with --apply to write")
|
||||
print()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -183,7 +183,8 @@ BANDS = [
|
||||
("roots", "Model roots", "How a term got here", "#0A1F44"),
|
||||
("ownership", "Ownership & Categorization", "Who is accountable", "#B8935A"),
|
||||
("business", "Business", "What the business means", "#8B3A3A"),
|
||||
("logical", "Logical", "What we commit to deliver", "#7A4E8B"),
|
||||
("logical", "Logical", "How the data is structured", "#7A4E8B"),
|
||||
("delivery", "Delivery", "What we promise, to whom", "#B8935A"),
|
||||
("physical", "Physical", "What actually runs", "#1E4B9B"),
|
||||
("consumption", "Consumption", "What people open in the morning", "#1E7A5F"),
|
||||
("context", "Actors & vocabularies", "Outside the object stack", "#6B6B6B"),
|
||||
@@ -193,6 +194,7 @@ LAYER_ROOT = {
|
||||
"pr:OwnershipLayerObject": "ownership",
|
||||
"pr:BusinessLayerObject": "business",
|
||||
"pr:LogicalLayerObject": "logical",
|
||||
"pr:DeliveryLayerObject": "delivery",
|
||||
"pr:PhysicalLayerObject": "physical",
|
||||
"pr:ConsumptionLayerObject": "consumption",
|
||||
}
|
||||
|
||||
@@ -0,0 +1,214 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
PR Data Meta Model - instance migration v1.0 -> v1.1
|
||||
=====================================================
|
||||
Rewrites instance files for the T-Box v1.1 vocabulary. Text-level rewriting on
|
||||
purpose: it preserves comments, ordering and formatting, which a parse-and-
|
||||
reserialise round trip through rdflib would destroy -- and those comments carry
|
||||
the arbitration notes.
|
||||
|
||||
USAGE
|
||||
python3 scripts/migrate_instances_v1_1.py # dry run, report only
|
||||
python3 scripts/migrate_instances_v1_1.py --apply # rewrite in place, .bak kept
|
||||
python3 scripts/migrate_instances_v1_1.py --apply f.ttl # a single file
|
||||
|
||||
WHAT IT DOES AUTOMATICALLY
|
||||
namespace /metamodel/0.9/ -> /metamodel/ (unversioned terms)
|
||||
belongsToDomain, belongsToSubDomain -> belongsTo
|
||||
implements -> represents
|
||||
hasSteward -> monitoredBy
|
||||
usesDataObject -> packages
|
||||
DataAsset -> DataInterface
|
||||
hasGrainDimension, hasGranularity -> flagged, not converted
|
||||
|
||||
WHAT IT REFUSES TO DO
|
||||
composedOf -> aboutConcept / usesConcept is NOT automated. Splitting it
|
||||
requires deciding which concept is the SUBJECT of each Business Object, and
|
||||
guessing that is exactly how v0.6 ended up with Calendar declared as being
|
||||
composed of Sell Out. The script lists the choices to be made and leaves
|
||||
composedOf in place until a human answers.
|
||||
|
||||
hasBusinessDefinition on a Business Object is likewise reported, not moved:
|
||||
the text must go to the Concept named by aboutConcept, which does not exist
|
||||
yet at the time this runs.
|
||||
"""
|
||||
import glob
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
INSTANCES = os.path.join(REPO, "instances")
|
||||
W = 78
|
||||
|
||||
# (pattern, replacement, label) -- applied in order, word-bounded
|
||||
RENAMES = [
|
||||
(r"https://ontology\.pernod-ricard\.com/metamodel/0\.9/",
|
||||
"https://ontology.pernod-ricard.com/metamodel/", "namespace 0.9 -> unversioned"),
|
||||
(r"\bpr:belongsToDomain\b", "pr:belongsTo", "belongsToDomain -> belongsTo"),
|
||||
(r"\bpr:belongsToSubDomain\b", "pr:belongsTo", "belongsToSubDomain -> belongsTo"),
|
||||
(r"\bpr:implements\b", "pr:represents", "implements -> represents"),
|
||||
(r"\bpr:hasSteward\b", "pr:monitoredBy", "hasSteward -> monitoredBy"),
|
||||
(r"\bpr:usesDataObject\b", "pr:packages", "usesDataObject -> packages"),
|
||||
(r"\bpr:DataAsset\b", "pr:DataInterface", "DataAsset -> DataInterface"),
|
||||
]
|
||||
|
||||
# things a human must decide, reported per file
|
||||
MANUAL = [
|
||||
(r"\bpr:composedOf\b", "composedOf",
|
||||
"Split into aboutConcept (exactly one -- the SUBJECT) and usesConcept "
|
||||
"(the rest). Not automated: guessing the subject is how v0.6 ended up "
|
||||
"with Calendar composed of Sell Out."),
|
||||
(r"\bpr:isComponentOf\b", "isComponentOf",
|
||||
"Inverse of the above. Re-express as usesConcept from the Business Object."),
|
||||
(r"\bpr:hasGranularity\b", "hasGranularity",
|
||||
"Grain moved off the Metric. Re-express as hasGrainElement on the Data "
|
||||
"Object, pointing at the Data Elements that make a row unique."),
|
||||
(r"\bpr:hasGrainDimension\b", "hasGrainDimension",
|
||||
"Same: hasGrainElement on the Data Object, targeting Data Elements."),
|
||||
]
|
||||
|
||||
|
||||
def subject_blocks(text):
|
||||
"""Yield (subject, block) for each top-level statement."""
|
||||
for block in re.split(r"\.\s*\n(?=\S)", text):
|
||||
m = re.match(r"\s*(\S+)", block)
|
||||
if m:
|
||||
yield m.group(1), block
|
||||
|
||||
|
||||
def business_objects_with_definition(text):
|
||||
"""Business Objects still carrying hasBusinessDefinition -- must move to the Concept."""
|
||||
out = []
|
||||
for subj, block in subject_blocks(text):
|
||||
if "pr:BusinessObject" in block and "pr:hasBusinessDefinition" in block:
|
||||
out.append(subj)
|
||||
return out
|
||||
|
||||
|
||||
def composed_of_map(text):
|
||||
"""Business Object -> list of concepts, so the subject can be chosen."""
|
||||
out = {}
|
||||
for subj, block in subject_blocks(text):
|
||||
if "pr:composedOf" not in block:
|
||||
continue
|
||||
m = re.search(r"pr:composedOf\s+([^;.]+)", block)
|
||||
if m:
|
||||
concepts = [c.strip() for c in m.group(1).split(",") if c.strip()]
|
||||
out[subj] = concepts
|
||||
return out
|
||||
|
||||
|
||||
def process(path, apply_changes):
|
||||
original = open(path, encoding="utf-8").read()
|
||||
text = original
|
||||
applied = []
|
||||
|
||||
for pattern, repl, label in RENAMES:
|
||||
text, n = re.subn(pattern, repl, text)
|
||||
if n:
|
||||
applied.append((label, n))
|
||||
|
||||
pending = []
|
||||
for pattern, label, advice in MANUAL:
|
||||
n = len(re.findall(pattern, original))
|
||||
if n:
|
||||
pending.append((label, n, advice))
|
||||
|
||||
bo_defs = business_objects_with_definition(original)
|
||||
comp = composed_of_map(original)
|
||||
|
||||
print("=" * W)
|
||||
print(os.path.relpath(path, REPO))
|
||||
print("=" * W)
|
||||
|
||||
if applied:
|
||||
print(" AUTOMATIC")
|
||||
for label, n in applied:
|
||||
print(" %-42s %4d occurrence(s)" % (label, n))
|
||||
else:
|
||||
print(" AUTOMATIC nothing to rewrite")
|
||||
|
||||
if pending:
|
||||
print(" MANUAL")
|
||||
for label, n, advice in pending:
|
||||
print(" %-42s %4d occurrence(s)" % (label, n))
|
||||
for line in _wrap(advice, 68):
|
||||
print(" " + line)
|
||||
|
||||
if bo_defs:
|
||||
print(" BUSINESS DEFINITIONS TO MOVE")
|
||||
print(" %d Business Object(s) still carry hasBusinessDefinition." % len(bo_defs))
|
||||
print(" Meaning belongs to the Concept. Move each text to the Concept")
|
||||
print(" named by that object's aboutConcept, then delete it here.")
|
||||
for s in bo_defs[:12]:
|
||||
print(" - %s" % s)
|
||||
if len(bo_defs) > 12:
|
||||
print(" ... and %d more" % (len(bo_defs) - 12))
|
||||
|
||||
if comp:
|
||||
print(" SUBJECT TO CHOOSE (composedOf -> aboutConcept + usesConcept)")
|
||||
for subj, concepts in list(comp.items())[:12]:
|
||||
print(" %s" % subj)
|
||||
print(" candidates: %s" % ", ".join(concepts))
|
||||
if len(comp) > 12:
|
||||
print(" ... and %d more" % (len(comp) - 12))
|
||||
|
||||
changed = text != original
|
||||
if changed and apply_changes:
|
||||
shutil.copy2(path, path + ".bak")
|
||||
open(path, "w", encoding="utf-8").write(text)
|
||||
print(" WRITTEN (backup at %s.bak)" % os.path.basename(path))
|
||||
elif changed:
|
||||
print(" DRY RUN -- re-run with --apply to write")
|
||||
print()
|
||||
return changed, len(pending) + len(bo_defs) + len(comp)
|
||||
|
||||
|
||||
def _wrap(text, width):
|
||||
words, line, out = text.split(), "", []
|
||||
for w in words:
|
||||
if len(line) + len(w) + 1 > width:
|
||||
out.append(line)
|
||||
line = w
|
||||
else:
|
||||
line = (line + " " + w).strip()
|
||||
if line:
|
||||
out.append(line)
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
apply_changes = "--apply" in sys.argv
|
||||
args = [a for a in sys.argv[1:] if not a.startswith("-")]
|
||||
files = ([a if os.path.isabs(a) else os.path.join(REPO, a) for a in args]
|
||||
or sorted(glob.glob(os.path.join(INSTANCES, "*.ttl"))))
|
||||
files = [f for f in files if os.path.exists(f)]
|
||||
|
||||
if not files:
|
||||
print("No instance file found.")
|
||||
sys.exit(2)
|
||||
|
||||
print()
|
||||
print("PR META MODEL - INSTANCE MIGRATION v1.0 -> v1.1 %s"
|
||||
% ("APPLY" if apply_changes else "DRY RUN"))
|
||||
print()
|
||||
|
||||
total_changed = total_manual = 0
|
||||
for f in files:
|
||||
changed, manual = process(f, apply_changes)
|
||||
total_changed += int(changed)
|
||||
total_manual += manual
|
||||
|
||||
print("=" * W)
|
||||
print("%d file(s) rewritten, %d decision(s) left to a human."
|
||||
% (total_changed, total_manual))
|
||||
if total_manual:
|
||||
print("Run the validator after the manual work: the XOR-of-meaning shape")
|
||||
print("on Data Elements will not pass until the concepts are wired.")
|
||||
print("=" * W)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,284 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
PR Data Meta Model - SHACL validation runner
|
||||
=============================================
|
||||
The automated gate behind rule DQ-011. Validates an instance graph against the
|
||||
executable rulebook (SHACL shapes), with the T-Box supplying subclass
|
||||
entailment.
|
||||
|
||||
USAGE
|
||||
python3 scripts/run_shacl_validation.py # all instances
|
||||
python3 scripts/run_shacl_validation.py instances/sodh.ttl # one file
|
||||
python3 scripts/run_shacl_validation.py --quiet # CI mode
|
||||
|
||||
Paths resolve against the repository root (the parent of scripts/), so the
|
||||
script runs correctly from anywhere.
|
||||
|
||||
EXIT CODES
|
||||
0 conforms, no blocking violation -> eligible for PUBLISHED
|
||||
1 violations found -> stays DRAFT / UNDER_REVIEW
|
||||
2 the run itself is not trustworthy -> a green result would be meaningless
|
||||
|
||||
Exit code 2 is the important one. A validation gate that passes because no
|
||||
shape matched anything is worse than one that fails: it reports success while
|
||||
checking nothing. The pre-flight below refuses to let that happen.
|
||||
|
||||
DEPENDENCIES
|
||||
pyshacl, rdflib. On GrosseBertha these need a venv built with the STABLE
|
||||
python (never the DSM system python, which moves between DSM releases):
|
||||
|
||||
/usr/local/bin/python3.9 -m venv venv
|
||||
. venv/bin/activate
|
||||
pip install pyshacl rdflib
|
||||
"""
|
||||
import glob
|
||||
import os
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
ONTOLOGY = os.path.join(REPO, "ontology", "pr_metamodel.ttl")
|
||||
SHAPES = os.path.join(REPO, "shapes", "pr_metamodel_shapes.ttl")
|
||||
INSTANCES = os.path.join(REPO, "instances")
|
||||
|
||||
W = 78
|
||||
BAR = "=" * W
|
||||
|
||||
|
||||
def rel(path):
|
||||
"""Path relative to the repo root, for readable output."""
|
||||
try:
|
||||
return os.path.relpath(path, REPO)
|
||||
except ValueError:
|
||||
return path
|
||||
|
||||
|
||||
def die(code, title, lines):
|
||||
print(BAR)
|
||||
print("PR META MODEL - SHACL VALIDATION")
|
||||
print(BAR)
|
||||
print(title)
|
||||
print("-" * W)
|
||||
for line in lines:
|
||||
print(" " + line)
|
||||
print(BAR)
|
||||
sys.exit(code)
|
||||
|
||||
|
||||
def load_deps():
|
||||
"""
|
||||
Import the validation stack.
|
||||
|
||||
Catches every exception, not just ImportError: a dependency that installs
|
||||
but cannot be imported (pyshacl built for a newer Python than the venv,
|
||||
an rdflib major-version clash) raises TypeError or AttributeError at import
|
||||
time. Letting those escape would end the process with an exit code CI reads
|
||||
as "the model has violations", when in fact validation never started.
|
||||
Anything wrong here is exit 2 -- the run is not trustworthy.
|
||||
"""
|
||||
try:
|
||||
from pyshacl import validate
|
||||
from rdflib import Graph
|
||||
from rdflib.namespace import RDF, SH
|
||||
return validate, Graph, RDF, SH
|
||||
except Exception as exc:
|
||||
hint = []
|
||||
if "unsupported operand type" in str(exc) or "|" in str(exc):
|
||||
hint = [
|
||||
"This is PEP 604 syntax (str | X) evaluated at import time:",
|
||||
"the installed pyshacl needs Python 3.10+, this venv is %d.%d."
|
||||
% (sys.version_info[0], sys.version_info[1]),
|
||||
"pyshacl advertises 3.9 support but its code does not honour it,",
|
||||
"so pip installs a version that cannot be imported.",
|
||||
"",
|
||||
"Pin a release that really supports this interpreter:",
|
||||
' pip install "pyshacl==0.26.0" "rdflib>=6.3.2,<7.1"',
|
||||
"or rebuild the venv on a newer Python if one exists.",
|
||||
]
|
||||
else:
|
||||
hint = [
|
||||
"Build the venv with the stable python, never the DSM system one:",
|
||||
" /usr/local/bin/python3.9 -m venv venv",
|
||||
" . venv/bin/activate && pip install pyshacl rdflib",
|
||||
]
|
||||
die(2, "DEPENDENCY UNUSABLE -- validation did not run",
|
||||
["%s: %s" % (type(exc).__name__, exc), ""] + hint)
|
||||
|
||||
|
||||
def term_namespaces(graph, predicate=None):
|
||||
"""
|
||||
Namespaces actually used by pr: terms in a graph, with usage counts.
|
||||
|
||||
Only real IRIs count. Literals are skipped: SPARQL constraint bodies
|
||||
(sh:select) embed their own PREFIX declarations, and counting those would
|
||||
put raw query text in the diagnostic table.
|
||||
"""
|
||||
counts = defaultdict(int)
|
||||
for s, p, o in graph:
|
||||
for term in (s, p, o):
|
||||
text = str(term)
|
||||
if "ontology.pernod-ricard.com/metamodel" not in text:
|
||||
continue
|
||||
if not text.startswith("http") or any(c in text for c in " \t\n\r<>\""):
|
||||
continue # literal, not an IRI
|
||||
counts[text.rsplit("/", 1)[0] + "/"] += 1
|
||||
return counts
|
||||
|
||||
|
||||
def preflight(data, shapes, onto, data_files):
|
||||
"""
|
||||
Refuse to produce a green result that means nothing.
|
||||
|
||||
Two ways a SHACL run passes while checking nothing:
|
||||
1. the shapes target classes in a namespace the instances never use;
|
||||
2. the ontology declares its classes in a third namespace, so subclass
|
||||
entailment cannot reach the instances and every shape targeting an
|
||||
abstract parent (MetaModelObject) finds zero focus nodes.
|
||||
Both look identical to a caller reading only "CONFORMS: True".
|
||||
"""
|
||||
problems = []
|
||||
ns_problem = False
|
||||
|
||||
if not len(data):
|
||||
problems.append("The instance graph is empty -- nothing to validate.")
|
||||
|
||||
ns_data = term_namespaces(data)
|
||||
ns_shapes = term_namespaces(shapes)
|
||||
ns_onto = term_namespaces(onto) if onto is not None else {}
|
||||
|
||||
if ns_data and ns_shapes and not (set(ns_data) & set(ns_shapes)):
|
||||
ns_problem = True
|
||||
problems.append(
|
||||
"Shapes and instances use disjoint term namespaces -- no shape can "
|
||||
"match any node.")
|
||||
|
||||
if ns_onto and ns_data and not (set(ns_onto) & set(ns_data)):
|
||||
ns_problem = True
|
||||
problems.append(
|
||||
"The T-Box declares its terms in a namespace the instances never "
|
||||
"use. Subclass entailment cannot reach the data, so every shape "
|
||||
"targeting an abstract parent finds zero focus nodes.")
|
||||
|
||||
if problems:
|
||||
lines = list(problems)
|
||||
lines.append("")
|
||||
lines.append("Namespaces in use:")
|
||||
for label, counts in (("instances", ns_data), ("shapes", ns_shapes),
|
||||
("ontology", ns_onto)):
|
||||
if counts:
|
||||
for ns, n in sorted(counts.items(), key=lambda kv: -kv[1]):
|
||||
lines.append(" %-10s %-52s %6d refs" % (label, ns, n))
|
||||
else:
|
||||
lines.append(" %-10s (none)" % label)
|
||||
if ns_problem:
|
||||
lines.append("")
|
||||
lines.append("Fix: align the namespaces before trusting any result.")
|
||||
lines.append(" sed -i 's|/metamodel/0.9/|/metamodel/|g' "
|
||||
"instances/*.ttl shapes/*.ttl")
|
||||
die(2, "PRE-FLIGHT FAILED -- a result here would be meaningless", lines)
|
||||
|
||||
|
||||
def main():
|
||||
args = [a for a in sys.argv[1:] if not a.startswith("-")]
|
||||
quiet = "--quiet" in sys.argv or "-q" in sys.argv
|
||||
|
||||
validate, Graph, RDF, SH = load_deps()
|
||||
|
||||
# ---- resolve inputs ---------------------------------------------------
|
||||
if args:
|
||||
data_files = [a if os.path.isabs(a) else os.path.join(REPO, a) for a in args]
|
||||
else:
|
||||
data_files = sorted(glob.glob(os.path.join(INSTANCES, "*.ttl")))
|
||||
|
||||
missing = [f for f in data_files if not os.path.exists(f)]
|
||||
if missing or not data_files:
|
||||
die(2, "NO INSTANCE DATA -- validation did not run",
|
||||
[rel(f) + " not found" for f in missing] or
|
||||
["instances/ contains no .ttl file"])
|
||||
|
||||
if not os.path.exists(SHAPES):
|
||||
die(2, "NO SHAPES -- validation did not run",
|
||||
[rel(SHAPES) + " not found"])
|
||||
|
||||
# ---- parse ------------------------------------------------------------
|
||||
data = Graph()
|
||||
for f in data_files:
|
||||
try:
|
||||
data.parse(f, format="turtle")
|
||||
except Exception as exc:
|
||||
die(2, "PARSE ERROR -- validation did not run",
|
||||
[rel(f), str(exc)])
|
||||
|
||||
try:
|
||||
shapes = Graph().parse(SHAPES, format="turtle")
|
||||
except Exception as exc:
|
||||
die(2, "PARSE ERROR in shapes -- validation did not run",
|
||||
[rel(SHAPES), str(exc)])
|
||||
|
||||
onto = None
|
||||
if os.path.exists(ONTOLOGY):
|
||||
try:
|
||||
onto = Graph().parse(ONTOLOGY, format="turtle")
|
||||
except Exception as exc:
|
||||
die(2, "PARSE ERROR in ontology -- validation did not run",
|
||||
[rel(ONTOLOGY), str(exc)])
|
||||
|
||||
preflight(data, shapes, onto, data_files)
|
||||
|
||||
# ---- validate ---------------------------------------------------------
|
||||
conforms, results, _ = validate(
|
||||
data_graph=data,
|
||||
shacl_graph=shapes,
|
||||
ont_graph=onto,
|
||||
inference="rdfs", # a Metric is also a MetaModelObject
|
||||
advanced=True, # SPARQL constraints (NR-002, OW-007, DQ-006)
|
||||
abort_on_first=False,
|
||||
meta_shacl=False,
|
||||
)
|
||||
|
||||
rows = []
|
||||
for res in results.subjects(RDF.type, SH.ValidationResult):
|
||||
sev = str(results.value(res, SH.resultSeverity)).rsplit("#", 1)[-1]
|
||||
focus = str(results.value(res, SH.focusNode)).rsplit("/", 1)[-1]
|
||||
msg = str(results.value(res, SH.resultMessage) or "")
|
||||
src = str(results.value(res, SH.sourceShape) or "").rsplit("/", 1)[-1]
|
||||
rows.append((sev, src, focus, msg))
|
||||
|
||||
violations = sum(1 for r in rows if r[0] == "Violation")
|
||||
warnings = len(rows) - violations
|
||||
|
||||
# ---- report -----------------------------------------------------------
|
||||
print(BAR)
|
||||
print("PR META MODEL - SHACL VALIDATION")
|
||||
print(" data : " + ", ".join(rel(f) for f in data_files))
|
||||
print(" shapes : " + rel(SHAPES))
|
||||
print(" ontology : " + (rel(ONTOLOGY) if onto is not None else "(absent - no entailment)"))
|
||||
print(" triples : %d data / %d shapes" % (len(data), len(shapes)))
|
||||
print(BAR)
|
||||
print("CONFORMS : %s" % conforms)
|
||||
print("RESULTS : %d total | %d BLOCKING | %d MAJOR (warning)"
|
||||
% (len(rows), violations, warnings))
|
||||
|
||||
if rows and not quiet:
|
||||
print("-" * W)
|
||||
by_shape = defaultdict(list)
|
||||
for sev, src, focus, msg in rows:
|
||||
by_shape[(sev, src)].append((focus, msg))
|
||||
# blocking first, then by volume: the rules to fix first come first
|
||||
order = sorted(by_shape.items(),
|
||||
key=lambda kv: (kv[0][0] != "Violation", -len(kv[1])))
|
||||
for (sev, src), items in order:
|
||||
tag = "BLOCK" if sev == "Violation" else "WARN "
|
||||
print("[%s] %-42s %3d node(s)" % (tag, src or "(unnamed shape)", len(items)))
|
||||
print(" %s" % (items[0][1][:66] or "(no message)"))
|
||||
for focus, _ in items[:6]:
|
||||
print(" - %s" % focus)
|
||||
if len(items) > 6:
|
||||
print(" ... and %d more" % (len(items) - 6))
|
||||
|
||||
print(BAR)
|
||||
sys.exit(0 if violations == 0 else 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user