Data & MoreEngineeringDeep dive 05

From a finding to a delete decision.

Classification is what turns a profiled document into an action. A profiler emits findings, tags mark and refine them, document classes raise them to the document level, and policies hand the result to a custodian who validates it and decides whether the document is kept, reclassified or deleted. This is how those layers fit together, and how the Elasticsearch indices behind them are shaped.

Detection finds it. People decide what happens to it.

1351detection algorithms
2320tags per tenant
45document classes per tenant
59policies per tenant
Section 01Overview

Five layers, bottom up

Classification is a layered refinement. Each layer is a stored query that reads what the layer below wrote onto the document, then writes its own marker. The chain runs from machine detection up to a human compliance decision. It does not end at a Microsoft Purview label: it ends at a policy, where a custodian validates the classification and decides retention or deletion. MIP labelling is one optional enforcement action, nothing more.

DETECT MARK CLASSIFY GOVERN VALIDATE Entry types profiler findings DS_EntryType_List Tags value-level markers DS_Tags Document classes document-level DS_DocumentClass Policy purpose + retention notification_settings scheduled_delete Custodian validates classification keep / DELETE a tag can emit a knowledge entry type if mip_enabled MIP / Purview label DS_Status = DELETED
Main data path Conditional / feedback Outcome / enforced state Written payload

The detection that produces entry types, the profiler running its algorithms, is covered in AI Profiler. This manual starts where the findings land.

Section 02The layers

What each layer does

Layer 01

Entry types

The detection findings written by the profiler: FULL_NAME, STREET_ADDRESS, S_LOC, and the knowledge family S_K_*. Each carries pii, sensitivity and a colour. They surface as DS_EntryType_List.

index entry_types · ~715 · dynamic:false
Layer 02

Tags

Stored queries, mostly over DS_Value. They stamp DS_Tags, can emit S_K_* entry types, and can mutate findings via flags: remove_labels, remove_values, rename_data, set_status, sticky.

index tags · ~2320 · dynamic:false
Layer 03

Document classes

Stored queries over DS_EntryType_List or DS_Tags. They answer "what kind of document is this?" and stamp DS_DocumentClass. Same engine as tags, different input and output field.

index document_classes · 45/tenant · dynamic:true
Layer 04

Policies

Group classified documents for a compliance purpose. Own the custodian report (notification_settings: misclassified tab, delete control, cycle) and retention (scheduled_delete, days_to_delete, duration). Optional channels: MIP, Gmail, GDrive.

index policies · ~59/tenant · dynamic:true
Tags and classes are the same engine at two altitudes. Both are stored queries that stamp an ID onto a document. They differ only in what they read (tags read values, classes read entry types or tags) and what they write (DS_Tags versus DS_DocumentClass). That is why they share the category model, the translation index, and the tagging task code.
Section 03Worked trace

"Religious Orientation", end to end

A real trace from one tenant. It shows a class that keys on tags, not entry types directly, and reaches the same classification by two independent routes: one driven by a detected entry type, one by literal German document phrases.

EVIDENCE TAGS CLASS GOVERN RELIGIOUS_ORIENTATION + FULL_NAME, not excluded pii 2 · sensitivity 4 · 3914 keywords DS_Value ~ "Kirchenaustritt" literal German phrases wildcard match, no entry type UN Religious Orientation veUeLZwBiRR6rBsVr02n entry-type route GER church resignation1 FOUeLZwBiRR6rBsVs07W keyword route Religious Orientation cat: Privacy Classification query: DS_Tags (A OR B) Privacy policy custodian report validate then DELETE
Match path To governance Stored query object

The class query is two OR'd blocks, each a single DS_Tags term. A document carrying either tag is classified, and the class ID is written to DS_DocumentClass. From there a privacy policy gathers the documents, a custodian reviews them, rejects false positives on the Misclassified tab, and the policy retention window drives deletion. The bridge is concrete: tag output (DS_Tags) is the class input.

Class query

DS_Tags = veU... (UN Religious)
  OR
DS_Tags = FOU... (GER church)

Section 04Index model

How the indices are structured

Every classification index shares a multi-tenant and subscription-provenance envelope, then adds its own fields. All tenants live in one index, separated by company_id; all rows are subscription-seeded from a shared baseline.

IndexCount / tenantDynamicDistinctive fields
entry_types~715falsename_normalized, method, keywords, redacted, pii, sensitivity, color
tags~2320falsequery, category_id, remove_labels/values/text, set_status, sticky, report_tag
document_classes45truequery, category_id, tag_document_class, corrective_actions
policies~59truequery, enforcement_*, scheduled_delete, days_to_delete, duration, notification_settings, mip_enabled
tag_category~200truename, active, profile_tag, show_in_treeview
document_class_category~21truename, active, show_in_dashboard
tag_names~3625truetag_id, type (tag|class), translated_field, ~40 lang columns
mip_label~30falselabel_id, name, level, tenant, color

Shared envelope

  • company_id the tenant key (keyword)
  • query the stored matching query (object)
  • status active toggle (categories use active)
  • order, risk, personal, profiled, total
  • duration / duration_start / duration_end
  • provenance: defined_from_subscriptions, subscription_source, original_subscription_id

Category & translation

Tags and classes are organised and localised by parallel side-indices.

Tag groupingtag_category
Class groupingdocument_class_category
Localisationtag_names (type tag|class)

tag_names is a single i18n index for both: roughly 2560 rows type:tag and 430 rows type:class per tenant.

Section 05Query grammar

One grammar for tags, classes and policies

The query object has the same shape everywhere: one or more blocks, each holding an array of filters. Blocks are OR'd; filters within a block are AND'd. This single grammar is why one matching engine can power all three query layers.

filter shape

{
  "field": "DS_EntryType_List",
  "type": "terms" | "wildcard",
  "compare": "equal" | "not_equal",
  "values": ["STREET_ADDRESS", ...]
}

  • field the document field to read: DS_Value, DS_EntryType_List, DS_Tags, DS_KnownPersons.
  • type terms for an exact set, wildcard for patterns.
  • compare equal or not_equal (the exclusion guard in the trace used not_equal).
  • Multiple blocks OR; multiple filters AND. That is the whole logic.
In one line

Classification ends at a person, not a label.

Detection feeds tags, tags feed classes, classes feed policies, and policies put the result in front of a custodian who validates it and decides keep, reclassify or delete. MIP labelling is an optional output, never the destination. Every layer is multi-tenant by company_id, subscription-seeded, and built on one stored-query grammar with shared category and translation layers.

support.dataandmore.com/en/knowledge/deep-dive/classification