Aperture: Overview of Proposed Changes

01Purpose and scope

This document describes the proposed evolution of Aperture from conventional trend analytics into a narrative-centric social-media intelligence system, including the target architecture, processing methodology, persistence changes, and implementation plan.

02Executive summary

Aperture will retain its existing cross-platform collection and asynchronous processing foundations while adding a batch-level narrative intelligence layer. Rather than treating all posts under a trend as one homogeneous dataset, the system will identify coherent lines of discussion and analyse them as distinct entities.

The scraper's collection size and the analytical batch size will be independently configurable. The initial target is approximately 10,000 events per day, with an initial internal batch size of 20 events. Jev will be the primary engine for batch sentiment and narrative classification. Gemini will be invoked when a batch has no defensible existing narrative match or is too mixed to classify reliably; it will group related unmatched batches before proposing new narrative candidates.

03Core concept: what is a narrative?

A narrative is a distinct, meaningful line of discussion occurring within a trend. It is not limited to a causal explanation, a factual claim, or a single opinion. It can describe how people frame an event, debate its legitimacy, discuss an associated issue, or talk about its consequences.

Example: “Protests in India”

These narratives do not all answer the same question. Some express a framing of the participants; some discuss the legitimacy of the protest; others focus on a connected social issue or consequence. They are separate lines of conversation within the same trend.

Shared keywords alone are insufficient. A post that mentions women's empowerment in passing should not automatically be assigned to the women's empowerment narrative. Classification must consider the meaning and relevance of the content.

04Product objectives and analytical outcomes

The narrative layer should make the broader trend more understandable, not replace trend-level analytics. A trend remains the broad subject or event being tracked; narratives represent distinct conversations inside or related to it.

05Current architecture baseline

The supplied backend uses Node.js/CommonJS, BullMQ, Redis, PostgreSQL, and a canonical collection format. The existing pipeline is intended to separate ingestion from analytics processing and final persistence.

Current worker responsibilities

The current producer validates canonical data from the Upstash stream and constructs a BullMQ flow. The existing database parent expects a fixed set of analytics child results, so introducing narrative outputs requires updating that contract carefully.

06Target processing architecture

The ingestion path should remain largely unchanged. The major addition is narrative-aware classification between canonical collection ingestion and final analytical persistence. The final BullMQ dependency graph will be based on the full code audit, especially the existing database parent's child-result and transaction handling.

SCRAPER
   |
   v
UPSTASH REDIS STREAM
   |
   v
PRODUCER: read, validate, and construct BullMQ flow
   |
   +---------------------+--------------------+-------------------+
   |                     |                    |                   |
   v                     v                    v                   v
Trend analysis    Demographic analysis  Network analysis   Jev analysis
                                                              |
                                                    Split collection into
                                                    configurable batches
                                                              |
                                                              v
                                                    Retrieve candidate pool
                                                              |
                                                              v
                                                    Batch narrative + sentiment
                                                              |
                                       +----------------------+------------------+
                                       |                      |                  |
                                       v                      v                  v
                                 Existing match          Mixed batch       No suitable match
                                       |                      |                  |
                                       |                      +--------+---------+
                                       |                               |
                                       |                               v
                                       |                     Gemini fallback
                                       |                     group and propose
                                       |                               |
                                       +---------------+---------------+
                                                       |
                                                       v
                                          Validated analysis results
                                                       |
                                                       v
                                           Database persistence parent
                                                       |
                      +-------------------------------+--------------------------+
                      |                               |                          |
                      v                               v                          v
              Canonical events                 Narrative records          Snapshots / metrics
                      |                               |                          |
                      +-------------------------------+--------------------------+
                                                       |
                                                       v
                                      Searchable narrative intelligence

07Batch-level classification strategy

The classification unit is a batch, not an individual event. This is intended to reduce redundant API requests, give Jev context across related posts, and avoid creating excessive niche narratives from small wording differences.

Configurable splitting

The current scraper may collect up to approximately 30 events per interval, but the analytical layer must not depend on that fixed value. If collection size changes, only configuration should need adjustment.

Jev's expected decision for each batch

1. Review the batch content and the bounded candidate narrative pool.
2. Identify the existing narrative that best represents the dominant line of discussion, if one does.
3. Assess relevance and whether the apparent match is sufficiently strong.
4. Determine whether the batch is mixed or lacks a defensible match.
5. Return batch-level sentiment characteristics using the agreed sentiment taxonomy.
6. Provide structured outputs that the backend can validate and persist.

08Mixed and unmatched batches

We agreed that the system must be able to abstain when a batch does not have a defensible dominant narrative. Forcing a winner in a mixed batch would create false precision and contaminate downstream analytics.

Illustrative outcomes

These are logical result categories; the exact persisted status names will be finalised with the schema. An uncertain model response and a technical API failure must remain distinct: uncertainty is a valid analytical result, while a timeout or service error is an operational failure that may require a retry.

09Relevance and event-assignment policy

Canonical events must be preserved even when they are not strongly relevant to a particular narrative. Their presence may still matter for understanding the trend collection and the broader conversation.

However, trend association and narrative evidence are not equivalent. A post may be collected because it relates to a trend without meaningfully participating in the batch's assigned narrative.

Where appropriate, the system can store an estimated relevant-event count or relevance measure. Such model-based values must be labelled as estimates, not verified ground truth.

10Jev's responsibilities

Jev is the primary structured reasoning and classification engine. It will handle decisions that can be expressed as typed questions against the current batch and candidate narrative context.

The implementation will use the TypeSafe SDK and documented response types. The exact primitives, schemas, and response validation will be verified against the installed SDK version and current official documentation during implementation.

11Sentiment architecture

The existing sentiment pipeline will be replaced by Jev as the production sentiment engine. The reason is cost efficiency and suitability for the required structured classification workflow; the user has already validated that Jev works for this purpose.

Separate analytical dimensions

These dimensions must not be collapsed into one label. A negative sentiment does not necessarily mean opposition to a narrative, and positive wording can be sarcastic. The sentiment taxonomy will be explicitly defined in Jev parameters.

The primary initial output is batch-level sentiment. Narrative-level sentiment should be calculated from content meaningfully associated with that narrative where the available evidence supports it. The system must not present a batch-level aggregate as if every post had that same sentiment.

Migration approach

1. Integrate Jev into the sentiment/narrative processing path.
2. Validate the output and compare a representative sample against the old pipeline as a sanity check.
3. Make Jev the production path.
4. Remove obsolete sentiment dependencies and modules once references and tests confirm they are no longer needed.

12Gemini fallback and narrative formation

Gemini is not the default classifier for every batch. Its purpose is to help discover coherent narratives when the existing narrative pool does not provide a defensible match or the batch is too mixed to classify reliably.

Fallback workflow

1. Collect candidates: gather unmatched or ambiguous batch results into a bounded workload. The system should not necessarily wait for the whole day; window size and trigger thresholds can be configured.
2. Group related content: Gemini identifies coherent groups of related unmatched batches rather than creating a narrative for every batch independently.
3. Propose narratives: each candidate includes a concise title, description, core line of discussion, associated batch IDs, and supporting evidence or ambiguity notes.
4. Validate output: validate schema, required fields, length limits, and referenced batch IDs before accepting the response.
5. Check for duplicates: compare candidates with existing global narratives to avoid creating near-duplicate identities.
6. Persist and reuse: approved narratives become available in the candidate pool for future Jev classification.

If no coherent grouping emerges, the batches remain unassigned or pending review rather than receiving invented narrative labels.

13Confidence, candidate pools, and cost

The candidate pool may contain many semantically related narratives. The system therefore needs to distinguish a model's preference for one candidate from evidence that the batch genuinely has a dominant narrative.

Decision principles

• Use a configurable decision threshold rather than hard-coding a number throughout the code.
• Start with a provisional threshold of 0.80 only if the selected Jev primitive exposes a compatible decision-strength or confidence measure.
• Do not interpret the threshold as a calibrated 80% probability of correctness without empirical validation.
• Consider ambiguity between candidates and relevance to the batch, not only the top-ranked candidate.
• Permit mixed or insufficient-evidence outcomes.

Token and request efficiency

The initial workload target is approximately 10,000 events per day. With a batch size of 20, that corresponds to roughly 500 classification batches per day if every event is processed once and there are no retries. This is a planning estimate, not a guaranteed request count or cost.

• Send compatible questions in one Jev request.
• Keep candidate pools bounded and relevant.
• Avoid repeating large context packages unnecessarily.
• Use Gemini only for unresolved batches.
• Measure requests, input tokens, latency, retries, and fallback rates.

Published TypeSafe pricing has been cited as $0.042 per million input tokens, with output tokens free under the referenced pricing model. Current pricing and the deployed account's terms should be rechecked during implementation. Real costs will depend on state size, candidate descriptions, prompt overhead, retries, and fallback volume.

14Conceptual data model

Narratives are global entities. They must not be children of a single trend in the relational model, because the same underlying line of discussion may occur in several trends. Cross-trend semantic clustering is a later feature, but the schema should not prevent it.

Trend 1 -----------+                         +----------- Trend 2
                   |                         |
                   +---- Narrative X --------+
                              |
                    +---------+----------+
                    |                    |
              Batch decisions       Event associations
                    |                    |
              Batch metrics        Relevant evidence
                    |                    |
                    +---------+----------+
                              |
                     Narrative snapshots

This is a conceptual model, not a final migration. Existing tables and keys must be inspected before deciding which entities are new and which existing tables should be extended.

Why keep batch and event associations separate?

The batch record captures the model's primary decision, preserving the agreed batch-first design. Event associations allow more precise evidence and sentiment calculations without asserting that every event in a batch belongs equally to its primary narrative. The distinction also makes results easier to audit.

15Persistence flow and database integrity

The existing database parent gathers analytics child results and coordinates persistence. The preferred approach is to retain that responsibility unless the full code audit identifies a strong reason to restructure it.

Expected persistence sequence

1. Validate the canonical collection and analytics child outputs.
2. Persist or resolve the trend and collection records.
3. Persist canonical events and required author records.
4. Resolve existing narrative identities and validate proposed new identities.
5. Persist batch-level classification decisions.
6. Persist narrative-to-trend and narrative-to-event associations where justified.
7. Persist sentiment and narrative snapshots using stable identifiers.
8. Mark the collection complete only when the required persistence operations succeed.

The precise transaction boundaries and ordering must match the existing schema and foreign-key relationships. We should not write narrative-event associations before their referenced event records exist.

Migration strategy

• Inspect the live schema and current database helper modules before writing DDL.
• Preserve existing trend, collection, event, and analytics data.
• Add constraints and indexes that support idempotent writes and common queries.
• Use explicit migration files or a repeatable migration process rather than relying solely on an old one-table bootstrap script.
• Keep queryable fields relational; use JSONB for flexible evidence or model metadata where justified.

16Candidate retrieval and global narrative search

Jev should not receive every narrative in the database on every request. As the global pool grows, context size and irrelevant candidates would increase cost and reduce decision quality.

Initial candidate retrieval

• Prioritise narratives already associated with the current trend.
• Allow relevant global narratives to be considered when they plausibly apply across trends.
• Apply a configurable maximum candidate count, initially in the approximate range of 30–50 as a starting hypothesis.
• Keep candidate descriptions concise and structured.
• Measure whether the correct narrative is present in the candidate set before tuning the limit.

The first implementation can use database-backed retrieval and stable matching metadata. More advanced semantic search or embeddings can be added later if the narrative library's size and measured retrieval quality justify it.

Longer-term capability

Narratives should be searchable entities in their own right. A researcher should eventually be able to search for a subject or line of discussion, find related narratives across multiple trends, inspect their supporting content, and compare their evolution over time. This is a future product capability; the initial schema should enable it without pretending full semantic clustering is already implemented.

17Snapshots, metrics, and analytics

Narrative analytics must be time-aware. A narrative is not just a label; it is an entity whose volume, sentiment, relevance, and prominence may change across collections and over time.

The existing trend snapshot and analytics tables should be reused where their semantics fit. Narrative-specific snapshots should be added only after reviewing existing snapshot conventions and uniqueness keys.

18Queue and worker integration

The current flow has sentiment, demographic, trend, and network children coordinated by a database parent. The narrative feature will change the sentiment processing path and extend the child-result contract.

Integration principles

• Keep canonical validation at the ingestion boundary.
• Keep analytics workers focused on producing structured results.
• Update the database parent's expected child-result set deliberately; a new child or combined Jev worker must not be silently omitted.
• Ensure a failed Jev request is not confused with a valid mixed classification.
• Decide whether Gemini fallback runs inside the classification workflow or in a separate queue based on latency, retry isolation, and throughput needs.
• Keep queue names, payload schemas, and worker responsibilities centralised and versioned where practical.

The final topology will be decided after inspecting queues.js, worker.js, producer.js, and the database modules together.

19Reliability, idempotency, and failure handling

The pipeline must remain correct when jobs retry, workers restart, API calls time out, or database transactions fail. Model calls are not guaranteed to execute exactly once, so persistence must be designed for idempotent reprocessing.

We should also inspect the current producer's polling and pending-message handling during the audit, including whether polling cycles can overlap and whether reclaimed stream messages are actually reprocessed. Any change there should be justified by the existing implementation rather than bundled into the feature without review.

20Validation, provenance, and analytical integrity

Narratives describe what content discusses; they are not automatically verified statements of fact. The system must avoid turning allegations, rumours, or a model-generated summary into asserted truth.

• Preserve links or stable identifiers back to the original events wherever available.
• Store the narrative's descriptive framing separately from factual verification.
• Validate all model output against an explicit schema and field limits.
• Validate that referenced batch IDs and event IDs belong to the expected collection.
• Keep model confidence/decision metadata separate from observed counts.
• Use cautious language in narrative summaries where the underlying content contains allegations or uncertain claims.
• Do not label an account a bot solely because its posts are similar or its engagement is unusual; such signals are not proof and bot detection is outside the initial core scope unless separately prioritised.

21Configuration and environment

The supplied config.js establishes required infrastructure environment variables for Upstash Redis, local Redis, and PostgreSQL. It exposes shared and worker-specific Redis connections, PostgreSQL connection helpers, health checks, and shutdown handling.

The narrative implementation should extend configuration in a controlled way, validating required settings at startup without exposing secrets in logs or reports.

Exact environment variable names and defaults will be finalised after inspecting package conventions and existing configuration patterns. API keys belong in environment configuration and must never be committed or copied into the report.

22Observability and operational metrics

Cost reduction and classification quality must be measured, not assumed. The implementation should make it possible to understand how the system behaves under real ingestion conditions.

These metrics will let the team tune batch size, candidate count, thresholds, fallback windows, and concurrency based on observed results rather than guesswork.

23Implementation plan and deliverables

Phase 1 — Code and schema audit

Inspect the complete worker/producer integration, queue definitions, database helper modules, package dependencies, current sentiment interface, and live schema/migrations.

Deliverable: exact file-level change list and confirmation of existing data/persistence contracts.

Phase 2 — Finalise the data model

Define global narrative identity, trend associations, batch decisions, event associations, snapshots, status values, unique constraints, and migration order.

Deliverable: reviewed schema and migration plan.

Phase 3 — Implement Jev classification

Build a standalone service that splits collections, retrieves candidates, asks compatible structured questions, validates results, and returns batch sentiment and narrative decisions without writing directly to the database.

Deliverable: Jev service, output contract, and unit tests.

Phase 4 — Implement Gemini fallback

Group related unmatched batches, generate structured narrative candidates, validate them, check for duplicates, and expose approved results for persistence.

Deliverable: fallback service and bounded retry/concurrency policy.

Phase 5 — Integrate queues and persistence

Update queue payloads, child-result validation, database parent logic, relational writes, snapshots, and processing status transitions.

Deliverable: end-to-end ingestion-to-persistence workflow.

Phase 6 — Retire the old sentiment path

Validate Jev output on a representative sample, move production processing to Jev, and remove obsolete sentiment code/dependencies after confirming there are no remaining consumers.

Deliverable: Jev-based production sentiment with obsolete path removed.

Phase 7 — Reliability and performance testing

Test edge cases, retries, idempotency, fallback behaviour, database consistency, and realistic load. Tune configuration using observed usage.

Deliverable: tested implementation, configuration documentation, and operational metrics.

24Testing and acceptance criteria

25Agreed decisions and design principles

26Items to verify during implementation

The conceptual direction is settled. The remaining items are implementation details that require the actual code and schema, rather than further high-level debate.

These are not blockers to the agreed architecture. They are details to finalise after reviewing the project files and validating the selected SDK interfaces.

27Proposed direction and next steps

Aperture will evolve from trend-level social media analytics into a narrative-centric intelligence system. The current ingestion pipeline remains the foundation; the key addition is a batch-oriented narrative and sentiment layer with a hybrid classification and discovery strategy.

Cross-platform sources
        ↓
Canonical events and collections
        ↓
Configurable internal batches
        ↓
Jev: dominant narrative + sentiment + relevance + ambiguity
        ↓
Existing narrative match ───────────────┐
        │                                │
        └─ Mixed / no suitable match     │
                    ↓                    │
             Gemini grouping             │
                    ↓                    │
          Validated narrative candidates │
                    └──────────┬─────────┘
                               ↓
                 Coordinated PostgreSQL persistence
                               ↓
           Global narratives + trend/event associations
                               ↓
          Narrative analytics, snapshots, and discovery

Immediate next step: audit queues.js, package.json, the database helper modules, existing schema/migrations, and the current sentiment interface. Then finalise the schema and implement the complete integrated change set, with tests and migration instructions.

Document status: proposed architecture and change overview based on the requirements and constraints discussed. Proposed schema objects, configuration names, and operational thresholds are explicitly provisional until verified against the current codebase.