Coniglio

Coniglio is Publica.la's event-tracking and session-analytics backend. It ingests millions of client-side tracking events daily, validates and stores them, then aggregates them into reading and listening sessions that other services (for example Farfalla) can query for analytics purposes.

Overview

Coniglio provides:

• HTTP ingestion endpoints for high-volume tracking events
• Validation, storage, and aggregation of events into user sessions
• Analytics data APIs for other services to consume
• Tenant-aware data isolation and aggregation logic
• Environment toggles to pause heavy aggregation during maintenance windows

Technology Stack

Layer	Technology
Runtime	PHP 8.3
Framework	Laravel 10
Database	SingleStoreDB (columnstore, sharded)
Queues	Redis/SQS (Horizon locally, SQS via Vapor in staging and production)
Testing	Composer + Pest
Static Analysis	PHPStan / Larastan, Pint
Observability	Sentry
Deployment	Laravel Vapor (serverless on AWS)

High-Level Data Flow

1. Ingestion – api/v1/track/* routes hit TrackEventHandler.
2. Validation – Controller dispatches ValidateTrackRequest job.
3. Event Dispatch – Job fires Laravel events (SessionStart, Heartbeat, …).
4. Storage – Listener TrackEventStorer inserts raw events into track_events.
5. Aggregation Scheduler – TrackEventsAggregator runs every minute (configurable).
   • Calculates a safe unprocessed time window (now-5 s).
   • Locks the range with RaceConditionMiddleware to avoid overlaps.
   • Creates a batch of TenantTrackEventsAggregator jobs (one per tenant).
6. Tenant Aggregation – Each job pages through events and feeds them to SessionProjector, creating or updating rows in sessions.
7. Data Access – Analytics endpoints provide access to session data for other services to query.

Environment Toggles

Variable	Purpose
TRACK_EVENTS_PROCESSING_AGGREGATION_ENABLED	Enable/disable aggregation jobs
TRACK_EVENTS_PROCESSING_SCHEDULED_AGGREGATION	Run aggregator via scheduler

Automatic detection in Kernel.php pauses aggregation every Tuesday 10:00–12:00 UTC, providing a maintenance window.

Local Development Quick-Start

cp .env.example .env && composer install
php artisan horizon            # or set QUEUE_CONNECTION=sync
php artisan schedule:work      # run scheduler
./vendor/bin/pest              # execute tests
# Optionally seed fixtures

Deployment

CI/CD is handled by GitLab:

1. .gitlab-ci.yml runs lint, static analysis, and tests.
2. Artifacts are deployed via Laravel Vapor using vapor.production.yml and vapor.staging.yml.

How It Scales

• Append-Only Writes – track_events table receives inserts only.
• Windowed Aggregation – Lock-based time windows eliminate per-row flags.
• Tenant Partitioning – Workload is split per tenant for linear scalability.
• Paged Processing – Aggregators paginate to keep jobs short-lived.
• Autoscaling – Horizon and Vapor balance worker counts to traffic; SingleStore columnstore optimises large analytic reads.

With this architecture Coniglio delivers reliable, near-real-time session analytics while maintaining linear throughput as event volume grows.

Related Systems

Coniglio provides analytics data to:

• Farfalla – which provides dashboards, reports, and exports by accessing Coniglio's database directly or through its API endpoints
• Medusa – triggers and insights for content automations
• Other services – ecosystem-wide analytics

---