Data & MoreEngineeringPlatform architecture

How data moves, source to enforced policy.

Seven layers carry a document from an external source through automated classification to policy enforcement and audit. External systems are crawled by source-specific connectors, indexed through the core API, enriched by a pool of workers, persisted in a shared store, watched by monitoring, and surfaced in the Vue client.

Connect, scan, classify, decide, act, report.

7layers, source to UI
~40services
1shared store: Elasticsearch
4transports: REST, SQL, AMQP, S3
Section 01The flow

The seven layers at a glance

Each layer hands off to the next through a named transport. Sources are crawled by connectors, connectors write to the core API and Elasticsearch, the API dispatches async work to the worker pool over RabbitMQ, and every layer reads and writes the shared store.

SOURCES CONNECTORS CORE API WORKERS STORES MONITORING FRONTEND M365 · Google Workspace · File shares · Azure AD · Websites ews · graph-ingestion · google · fileshare-service · collector · website-source api :8000 · task-worker · iam :5000 · dlp :5050 · chat-api classification · ai-profiler · enforcement · tagging · ocr · DSM · sync · backup Elasticsearch :9200 · PostgreSQL :5432 · RabbitMQ :5672 · AWS S3 alerts · py-dm-notify · analytics · flower · metricbeat client (Vue 3) · chat-client · nginx API calls, OAuth, incremental sync HTTP batch ingest, direct ES index Celery tasks over AMQP :5672 REST :9200, SQL :5432, S3 API health queries, metrics, notifications HTTP / JSON REST, JWT via IAM
Hand-off between layers Processing / connector tier Edge or store tier
Layer 01External data sourcesOrigin of customer data to be governed
Microsoft 365
Outlook / SharePoint / OneDrive / Teams
Primary enterprise source: email, documents, file storage and team chat.
proto EWS / Graph API · auth OAuth 2.0 / Azure AD
Google Workspace
Gmail / Drive
Workspace mail and drive content.
proto Google APIs · auth OAuth 2.0 / service account
Network fileshare
SMB / NFS
Files, directories and permissions on shares.
Azure AD
Users / Groups
Identities and group memberships.
proto Microsoft Graph API
Websites
HTTP crawl
Published web pages and content.
API calls and crawlingOAuth tokensIncremental sync
Layer 02Connector servicesSource-specific ingestion adapters
ews
Java / Spring Boot
Exchange Web Services for Outlook, OneDrive, SharePoint, Teams. Crawl, move, delete, validate, revert.
out Elasticsearch
graph-ingestion
Scala / Maven
MS Graph with graph-management, dm-graph-ingestion and graph-enforcer.
connects ES + RabbitMQ
google
Python
Gmail and Google Drive scanning.
out Elasticsearch
fileshare-service
Java
Network file scanning over SMB / NFS.
out Elasticsearch
collector
Java / Gradle
Universal batch collector for generic source types.
out API then Elasticsearch
website-source
Python
HTTP / HTTPS web crawling.
out ES + RabbitMQ
HTTP batch ingestionDirect ES indexing
Layer 03Core API layerCentral orchestrator and auth
api
Python / Flask :8000
REST API with 20+ blueprint modules: documents, sources, policies, tags, reports, users, config.
connects ES, PostgreSQL, RabbitMQ, IAM
task-worker
Python / Celery
Async job processor via the RabbitMQ broker.
broker RabbitMQ :5672 · monitor Flower :5555
iam
Python / Flask :5000
Identity and access with JWT, company-scoped isolation.
connects PostgreSQL
dlp
Python / Flask + Celery :5050
Data loss prevention with a dedicated PostgreSQL database (dlp_dam).
connects ES, PostgreSQL, RabbitMQ
chat-api
Python
LLM-powered document Q&A for chat-based compliance.
connects ES, IAM, llm-management
Celery tasks via AMQP :5672Periodic scheduled jobs
Layer 04Processing workersCelery tasks consumed from RabbitMQ
Classification
Celery task
PII detection, sensitivity scoring, language identification.
out DS_PiiScore, DS_SensitivityScore
ai-profiler
Python / spaCy / Celery
NLP entity extraction and document profiling.
out ES profiler results
Policy enforcement
Celery task
Retention rules: delete, archive, quarantine with full audit.
audit deleted_data ES index
Tag assignment
Celery task
Delta tagging and document-class assignment.
out DS_Tags, DS_DocumentClass
ocr
Python
Text extraction from scanned documents and images.
out enriched document in ES
DataSubject Manager
Java
VIP and entity matching against the known-persons database.
out ES data-subject metadata
ES / PG sync
Celery task
Bidirectional Elasticsearch to PostgreSQL synchronisation.
tasks es2postgre, syncpostgres
Backup worker
Celery task
Daily S3 exports as JSON batches (750 docs/file).
target S3, 180-day retention
REST :9200SQL :5432AMQP :5672S3 API
Layer 05Data storesPersistence and messaging infrastructure
Elasticsearch 9.x
Primary index :9200
Document search, classification data, audit logs and system events. The central store of the platform.
datasourcesaccountstags policiesdeleted_dataanalyticsalgorithms document_classeslogshistory
version 9.3.1 · backup monthly SLM snapshots to S3
PostgreSQL 17.x
Relational state :5432
Users, configuration, DLP state, IAM accounts, PowerBI datasets and vector embeddings.
defaultdlp_damiampowerbivector
version 17.6 (ECR) · volume postgres_data
RabbitMQ 4.x
Message broker :5672
Celery task queue over AMQP. Management UI on :15672.
version 4.2.2 · volume rabbitmq_4
AWS S3
Backup storage
Daily JSON index exports per company / workspace plus monthly ES snapshots.
retention 180 days, auto-cleanup
Health queriesEmail notificationsMetrics collection
Layer 06Monitoring and alertingObservability and system health
Management Tool, Alerts
Python / 40+ alert types
Monitors ingestion speed, policy progress, ES index health, certificate expiry and Azure sync.
connects ES, RabbitMQ
py-dm-notify
Python
Email delivery for compliance notifications and reports.
analytics
Python / Flask
Compliance metrics aggregation and KPI reporting.
connects Elasticsearch
flower
Web UI :5555
Real-time Celery worker and task dashboard.
metricbeat
Elastic Stack
Collects system and container metrics for monitoring ES.
HTTP / JSON RESTJWT auth via IAM
Layer 07Frontend and user interfaceUser-facing applications
client
Vue 3 / TypeScript / Vite :9000
Primary web app: document search, classification management, policy administration, reports and case logs.
stack Vue 3, TS, Vite, Vuex, Chart.js
chat-client
Frontend
LLM chat interface for document Q&A.
backend chat-api
nginx
Reverse proxy :443 :80
SSL termination via Let's Encrypt. Routes / to client, /api to api, /iam to iam.
version 1.24.0
Section 02Document lifecycle

One document, one path

Source→ Connector crawls→ API indexes to ES→ Workers classify→ Tags and policies applied→ Policy enforcement→ Audit trail→ S3 backup
Section 03Key connections

Who talks to whom

API ↔ ESREST :9200
API ↔ PostgreSQLSQL :5432
API → RabbitMQAMQP :5672
Workers ↔ ESREST :9200
Workers → PostgreSQLSQL :5432
Workers → S3AWS SDK
Client → APIHTTP :8000
API → IAMJWT :5000
NGINX → all:443 reverse proxy
Docker networkdm, internal DNS
Data & More ApS. Generated from a dm_main repository exploration. Multi-tenant SaaS, roughly 40 services on Docker Compose. Conceptual layering simplified for legibility; not a literal deployment topology.