The Context Graph
The most valuable fact is usually the one the file does not contain.
AI can increasingly describe what a file visibly, audibly or textually contains. People create durable value by adding what was actually true, why it happened, what changed, who knows, what cannot be seen, whether it worked, how it was verified, how the asset changed and what use is permitted.
LOWER MARGINAL VALUE
“A man repairing a wooden chair.”
A capable model can usually recover a surface description.
GROUNDED HUMAN CONTEXT
The recorder is the maker. Moisture exposure caused a 12-year-old white-oak mortise-and-tenon joint to fail. Moisture measured 18%. The repair used polyurethane glue because the hide glue had deteriorated. It remained stable after 18 months.
Marginal data value ≈ non-inferability × truthfulness × causal specificity × rarity × granularity × linkability × verification quality
How Sally represents it
A caption remains useful for search, but Sally stores human knowledge as a graph so a weak guess never looks equivalent to a measured or independently verified fact.
Asset
Fingerprint, original/derivative status, creator, purpose, rights and related assets.
Anchor
The exact image region, time segment, text span or 3D component a claim concerns.
Entity
A stable person, animal, object, machine, place, organization, work or project identity.
Event
Participants, roles, initial state, action, tool, resulting state, measurement and outcome.
Assertion
Claim, annotator, knowledge basis, confidence, evidence, alternatives and verification.
Provenance
Creation, import, editing, transformation, verification, parent asset and rights history.
Precise anchors
Whole assetPhoto region, polygon, mask, box or keypointVideo or audio time segmentDocument character, paragraph or page span3D component or mesh IDPersistent entityEvent
A small packet first
Nobody should complete fifty fields for every item. The editor starts with ten medium-specific questions, saves partial answers, and reveals specialist prompts only when they are useful.
Photographs
- Why was this taken?
- Who or what is shown, using stable identities and roles?
- What happened immediately before?
- What happened afterward?
- What important fact is not visually apparent?
- What measurements or external verification exist?
- Who took it, and what was their relationship to the event?
- Is it original or edited?
- What is uncertain?
- What use is permitted?
Videos
- What was the goal?
- Who acted on what?
- Where does each meaningful step begin and end?
- What state changed?
- Did it succeed?
- What failed or required correction?
- What happened after the clip?
- What sensors or measurements can be linked?
- Was it natural, staged or edited?
- What use is permitted?
Audio recordings
- What physically produced each sound?
- Who spoke, and when?
- What exactly was said?
- What did the speaker mean?
- Who was being addressed?
- What happened in response?
- What was the acoustic setting?
- Was the recording processed or edited?
- What is uncertain?
- What voice and usage permissions apply?
Written documents
- Why was it written?
- Who actually contributed, and in what roles?
- What decision or action was it intended to support?
- Which claims are important?
- What evidence supports each claim?
- What later proved true or false?
- What period, jurisdiction or situation does it apply to?
- What version is this, and why did it change?
- What is uncertain or disputed?
- What access and use restrictions apply?
Personal creations and physical artifacts
- What was the creator trying to make or communicate?
- What constraints governed the design?
- What exact materials were used?
- What tools and settings were used?
- What was the step-by-step process?
- What decisions and alternatives arose?
- What failed or had to be corrected?
- What are the dimensions and physical properties?
- How did the result perform or how was it received?
- What source files, intermediate stages and rights are available?
Everything else
- Why does this asset exist?
- Who or what does it concern?
- What was true before it?
- What action or change occurred?
- What was true afterward?
- What hidden fact matters most?
- What evidence or measurement supports the account?
- Who knows this and how?
- What remains uncertain?
- What use is permitted?
Complete ranked vocabulary
These prompts are ordered by the human information they add beyond a capable model's own extraction. The lowest tier is still searchable, but normally should be automated.
Photographs
What the image cannot prove — Transformative human context
- Verified external ground truth
- Causal history
- Before-and-after state
- Capture intent
- Persistent subject or object identity
- Roles and relationships
- Hidden physical properties
- Quantitative measurements
Interpretation and missing context — Very high value
- Expert interpretation and rationale
- Non-visible context
- Exact event context
- Longitudinal state link
- Negative and disambiguating facts
- Selection context
- Authenticity and edit history
- Recorder firsthand status
Spatial and functional grounding — High value
- True scale and geometry
- Region-level box, polygon, mask or keypoint
- Object relationships
- Occlusion and visibility status
- Functional state and affordance
- Subject self-report, explicitly labeled
Recorder and capture context — Supporting context
- Recorder stable ID, role, expertise and relationship
- Camera, lens, focal length, aperture, shutter, ISO, focus and stabilization
- Lighting source, direction, temperature, flash and reflectors
- Color, lens, sensor and geometry calibration
- Blur, glare, compression, clipping, noise and rolling-shutter limitations
- Content restrictions, privacy zones and redaction history
Usually inferable — Usually machine-derivable
- Generic caption — capture automatically unless a human is correcting an error.
- Dominant colors
- Broad aesthetic style
- Generic emotional tone
- Basic object keywords
Videos
Action, state change and outcome — Transformative human context
- Goal → action → state change → outcome graph
- Objective success or failure and evidence
- Temporal step segmentation
- Object state before and after each action
- Errors, near misses and recovery actions
- Actor intent and decision rationale
- External outcome after the video ends
- Synchronized force, torque, touch, pose, control, gaze, depth, IMU and sensor data
Interaction and causal reasoning — Very high value
- Persistent actor, object and location identities
- Hand-object or tool-object contact
- Affordances and constraints
- Causal links rather than temporal coincidence
- Counterfactual explanation
- Participant roles and relationships
- Speaker-to-face and speaker-to-action mapping
- Time-aligned transcript with resolved pronouns and references
- Think-aloud narration
- Safety state, hazards, protective equipment and near-accidents
- Expert quality assessment
Spatial and multimodal context — High value
- 3D scene layout, camera trajectory, object poses and scale
- Multicamera synchronization
- Audio-video correspondence
- Gaze and attention
- Occluded action with evidence source
- Environment and constraints
- Object material and physical properties
- Reported and measured effort and difficulty
Recording provenance — Supporting context
- Recorder identity, role, expertise and participant relationship
- Natural, staged, scripted, reenacted or selectively edited status
- Why recording started and stopped
- Missing intervals and reasons for gaps
- Cuts, slow motion, compression, stabilization, overlays and generated frames
- Frame rate, exposure, rolling shutter, lens and calibration
- Consent by participant and segment
- Ambiguities and annotator disagreement
Usually inferable — Usually machine-derivable
- Whole-video generic summary — capture automatically unless a human is correcting an error.
- Broad activity class
- General scene category
- Generic mood or cinematic style
- Resolution, duration and encoding
Audio recordings
Source, meaning and response — Transformative human context
- Physical source ground truth
- Causal action and source state
- Time-aligned verbatim transcript with disfluencies and uncertainty
- Speaker or source diarization
- Intended meaning and resolved referents
- Conversational or functional intent
- Outcome or listener/system response
- Before-and-after context
Speech and source context — Very high value
- Speaker roles, relationships and expertise
- Firsthand, quoted, read, rehearsed or repeated knowledge
- Overlapping speech and non-speech events
- Clean isolated source or stem
- Source location, distance, direction and movement
- Occlusion and intervening objects
- Self-reported emotion, fatigue, pain, urgency or cognitive state
- Language, dialect, accent, native language, code-switching and pronunciation
- Proper-noun spelling and entity resolution
- Nonverbal vocal events
- Meaningful silence and interruption
- Confidence, ambiguity and alternate transcripts
Physical and acoustic context — High value
- Room or environment type, dimensions, materials and reverberation
- Weather, wind, temperature and atmosphere
- Machine speed, load, temperature and maintenance state
- Contact material
- Exact onset, offset and repeating pattern
- Microphone type, polar pattern, placement and orientation
- Array geometry and synchronization
- Gain, AGC, limiter, clipping and calibration
- Sample rate, bit depth and channels
Audio provenance and rights — Supporting context
- Recorder identity, role and purpose
- Original, reposted or re-recorded status
- Noise reduction, EQ, pitch, time, compression and enhancement
- Splicing, omitted intervals and generated or cloned voices
- Transcript creator, correction and verifier
- Voice-use consent, impersonation restrictions and commercial permissions
Usually inferable — Usually machine-derivable
- Broad topic — capture automatically unless a human is correcting an error.
- Generic mood
- General genre
- Unverified automatic transcript
- Simple speech, music or noise label
Written documents
Truth, evidence and outcome — Transformative human context
- Claim-level truth status with assessment date
- Evidence linked to each claim
- Subsequent real-world outcome
- Document purpose and decision context
- Applicability boundaries
- Methodology and calculation lineage
- Version history and change rationale
- Author and contributor roles, including AI contribution
Interpretation and applicability — Very high value
- Author expertise, organizational role and subject relationship
- Intended audience and assumed knowledge
- Definitions and acronyms
- Entity resolution
- Confidence for each assertion
- Competing interpretations or dissent
- Fact, opinion, hypothesis or recommendation
- Quote, paraphrase or original analysis
- Satire, sarcasm, exaggeration or hypothetical status
- Current, historical, proposed or predicted status
- Whether instructions were followed
- Omissions, redactions and reasons
Institutional provenance — High value
- Supersedes, amends, responds to, cites, contradicts or implements another document
- Draft, final, approved, signed, withdrawn or archived status
- Approver and authority
- Review history and unresolved comments
- Governing law, policy, regulation or standard
- Confidentiality and permitted audience
- Retention and destruction requirement
- Original, copy, scan, transcription or OCR status
- Human-written, AI-assisted or AI-generated portions
- Translation source, translator and review status
Document structure — Supporting context
- Reading order and heading hierarchy
- Table row, column and merged-cell semantics
- Figure-caption-claim links
- Footnote and endnote targets
- Signature and initial locations
- Handwritten edits and author
- Page omissions or scanning defects
- Language variety and domain vocabulary
- Text-span privacy or privilege markings
Usually inferable — Usually machine-derivable
- Generic summary — capture automatically unless a human is correcting an error.
- Broad topic
- Sentiment
- Reading level
- Generic keywords
- Creator and date already embedded
Personal creations and physical artifacts
Production graph — Transformative human context
- Complete process sequence with intermediate assets
- Creator intent
- Design requirements and constraints
- Exact materials, grades, batches, sources and condition
- Material properties at creation time
- Tools and process parameters
- Decision rationale
- Failures, mistakes and rework
- Verified performance or reception
- Source files and editable structure
Physical and process detail — Very high value
- Dimensions, scale and tolerance
- Component and assembly relationships
- Hidden joints, armature, fasteners, adhesives and internal structure
- Intermediate states and discarded versions
- Expected versus actual result
- Rejected alternatives and why
- Human, collaborator and AI contribution breakdown
- References, inspirations and permissions
- Time by stage
- Creation place and workshop conditions
- Feel, balance, temperature, weight, texture, flexibility, smell and sound
- Maintenance, aging, damage and restoration
- Version, edition, prototype or uniqueness status
- Creator stable ID, expertise and cultural context
Ownership, authenticity and interpretation — High value
- Ownership and custody history
- Exhibition, publication, sale and commission history
- Signature, maker marks and serial numbers
- Authenticity evidence
- Restoration and modification events
- Intended display, lighting, orientation and distance
- Symbolism and personal meaning
- Cultural technique/tradition and source qualification
- Audience critique separated from creator intent
- Rights, licensing, attribution and AI permissions
Usually inferable — Usually machine-derivable
- Broad style — capture automatically unless a human is correcting an error.
- Dominant palette
- Generic subject description
- Aesthetic mood
- Generic handmade or digital label
Everything else
Ground truth and change — Transformative human context
- Verified real-world outcome
- Cause and intervention
- Before-and-after state
- Creator purpose or question
- Persistent identities, roles and relationships
- Hidden physical or experiential properties
- Expert interpretation with rationale
- Failures, anomalies and negative evidence
- Uncertainty and competing explanations
Grounding and provenance — Very high value
- Provenance and transformation history
- Exact temporal and spatial context
- Cross-asset relationships
- Sampling and selection context
- Measurements and calibration
- Annotation source and confidence
- Creator and annotator qualifications
- Technical settings
Usually machine-derivable — Usually machine-derivable
- Generic description and keywords — capture automatically unless a human is correcting an error.
Specialist creation packets
A finished rendering or object hides the production graph. Sally can preserve the editable structure, materials, parameters, decisions, failures and validation behind it.
Drawing Painting
- Intended subject, symbolism, narrative, focal point and viewing order
- Reference provenance, compositional grid, perspective and deliberate distortion
- Substrate, primer, paper/canvas, pigment, binder, solvent and varnish
- Archival/material compatibility and drying or curing conditions
- Underdrawing, layer/glazing order, tools, stroke direction, pressure and dilution
- Drying, sanding, scraping, masking, corrections, overpainting and final coating
- Layer hierarchy, blend modes, masks, brush engine, stylus pressure and tilt
- Vector stroke order, undo/redo, selections, transforms and reference layers
- Color profile, canvas, software, version and plug-ins
- AI-edited regions, prompts, model and version
3d Model
- Original meshes, procedural graph, topology and polygon count
- Coordinate system, scale, units, normals, smoothing and subdivisions
- UVs, sculpt layers, retopology, collision geometry and LOD variants
- Material assignment, shader graph, texture provenance and physical parameters
- Scene graph, hierarchy, camera, lens, lighting, exposure and color management
- Render engine, sampling, denoising and bounce limits
- Rig, skeleton, skin weights, constraints, keyframes and motion-capture source
- Mass, friction, elasticity, gravity, cloth/fluid/particle settings and random seed
- Software, plug-ins, scanned versus modeled geometry and editing history
- Human/generated components and license of every incorporated asset
Woodworking
- Species, board/blank ID, source, drying method and moisture content
- Grain/ring orientation, figure, knots, checks, defects, density and hardness
- Sketch/CAD, dimensions, tolerances, load, joinery and wood-movement rationale
- Indoor/outdoor exposure and workshop temperature/humidity
- Tool model, blade/bit/gouge, speed, feed, toolpath, depth and cut direction
- Sharpening angle, sharpness and dust extraction
- Joint, adhesive, fastener, pilot hole, clamp pressure and duration
- Sanding progression and complete finishing schedule
- Tear-out, burning, chipping, movement, warping, failure and repair
- Load test, long-term stability, maintenance and customer outcome
Metal Fabrication
- Alloy, grade, certification/heat number, stock, temper and prior treatment
- Cutting method, blade/abrasive, gas, current, speed, kerf and pierce settings
- Bend angle/radius, tooling, grain, springback, tonnage and forming sequence
- Welding process, base/filler, electrode, shielding gas and flow
- Current, voltage, wire feed, travel, polarity, torch angle and stick-out
- Joint prep, root gap, bevel, tack/pass sequence, preheat and heat input
- Machine, tool/insert, coating, speed, feed, cut, coolant and workholding
- Tool wear, surface finish, tolerance and inspection method
- Distortion, porosity, undercut, fusion, cracking, penetration and hardness
- NDT, tensile/fatigue/load result, failure location, rework and corrosion
Specialist audio and documents
Audio: speech
- Speaker, addressee, roles and relationship
- Firsthand versus quoted knowledge
- Intent, goal, referents, turn-taking and interruption
- Sarcasm, irony or humor and correct names/technical terms
- Agreement, understanding, response and later factual accuracy
Audio: music
- Composer, performer, instrument model/material and tuning
- Key, meter, tempo changes and aligned score/MIDI
- Note timing, articulation, dynamics and improvisation
- Sections, takes, stems, microphone layout and acoustics
- Intent, errors, corrections and composition/performance/sampling/AI rights
Audio: mechanical
- Machine, part number, material and operating state
- Speed, load, temperature, lubrication and maintenance
- Known defect and teardown confirmation
- Repair and whether the sound disappeared
- Source geometry, background machinery and normal baseline
Document: legal
- Jurisdiction, court/agency, posture and controlling/persuasive authority
- Effective, amended, overruled and superseded status
- Definitions, obligations, exceptions, conditions, deadlines and remedies
- Waiver and mandatory/discretionary language
- Provision proposer, negotiation history and enforcement outcome
Document: scientific
- Hypothesis, preregistration, study design and sample criteria
- Instruments, raw data and analysis code
- Negative results, deviations, confounders and adverse events
- Replication, external validity, corrections and retractions
Document: business
- Decision, owner, options, assumptions and KPI definition
- Baseline, target, implementation date and actual result
- Variance, beneficiaries/harms, reversibility and lessons learned
Document: personal
- People, relationship, event and preceding context
- Self-reported emotion rather than inferred emotion
- Contemporaneous versus retrospective status
- Memory, speculation or fact and later outcome
- Personal-retrieval-only and other privacy restrictions
Evidence and confidence stay separate
A source can be highly confident and wrong. Each assertion records both confidence and the basis for knowing it.
| Reliability | Knowledge basis |
|---|---|
| 5 | Instrumentally or independently verified |
| 4 | Qualified expert directly examined it |
| 3 | Firsthand direct observation |
| 2 | Subject or participant self-report |
| 1 | Secondhand report |
| 0 | Unverified inference |
Permission is a gate, not a footnote
Rights do not make a record more truthful, but they determine whether anyone may use it. Unknown never silently means yes, and recording permission never publishes an item.
- Creator/copyright ownership
- Consent of identifiable participants
- Personal search
- Personal analysis
- Model training
- Fine-tuning
- Generation
- Commercial deployment
- Public display
- Derivatives and synthetic outputs
- Attribution requirement
- Geographic or time restriction
- Sensitive-data classification
- Revocation or deletion process
- Minor, health, private-location or confidential restrictions
Context arrives over time
Delayed outcomes are unusually rare and useful, so Sally separates creation, immediate recollection and later verification.
At creation
Creator or recorder · Purpose, time and place · Participants and firsthand status · Device/original and initial consent · Immediate physical context
Immediately afterward
What happened and why · Intent, hidden context and measurements · Difficulties, errors and subjective experience · Initial assessment
After enough time
Verified outcome, diagnosis and durability · Repair success or recurrence · Forecast accuracy or user reaction · Expert verification · Changed rights or consent
Provenance helps assess a claim; it does not make it true
Sally keeps creator, annotator and verifier separate. A signature can show who made a statement and whether it changed, but factual truth still depends on evidence.
The model draws on established ideas from IPTC Photo Metadata, W3C PROV, C2PA, Dublin Core, and CIDOC CRM. Sally's schema is its own pragmatic personal-data model, not a claim of certification.