Sentinel — Point-in-Time Market Research OS

A point-in-time market research engine for Indian equities, built to resist self-deception rather than predict the market.

---

1. Introduction

In financial engineering and quantitative research, the easiest person to fool is yourself.

The retail algorithmic trading space is saturated with overfitted backtests showcasing astronomical Sharpe ratios and hockey-stick equity curves. Yet almost all of them collapse into immediate losses when deployed live. The culprit is almost never bad math; it is lookahead bias and data leakage—models quietly learning from future earnings revisions, post-hoc corporate restatements, or survivorship-biased universes that were physically unknowable on the day of trading.

Sentinel is an institutional-grade, point-in-time market research operating system built specifically for Indian equities (NSE/BSE). It is not a stock-picking signal service, not a Telegram alert channel, and not an automated get-rich bot. It is rigorous research infrastructure designed with one primary directive: make it mathematically impossible to fool yourself with a backtest.

---

2. The Problem

Backtesting on historical financial data suffers from catastrophic subtle leaks:

1. Restatement Bias: A corporate earnings report published on October 15th may be revised downward on December 2nd due to an audit. If a model training on November 1st accesses the revised number, it is cheating by reading a future audit.
2. Knowledge Date vs. Event Date Discrepancy: GDP or inflation numbers announced at 5:30 PM after market close cannot be traded at 3:15 PM on that same calendar date. Failing to model publication latency produces imaginary profits.
3. Survivorship Bias: Testing strategies exclusively on companies currently in the NIFTY 50 ignores the dozens of firms that went bankrupt or were delisted over the last decade.
4. P-Hacking & Vanity Metrics: Backtesting 1,000 parameter combinations until one achieves a 3.0 Sharpe ratio is statistical noise disguised as alpha.

---

3. Why Existing Solutions Fail

• Standard Python Quant Libraries (Backtrader, Zipline forks): Assume clean price matrices and leave point-in-time data hygiene entirely to the researcher. They do not prevent future-looking column joins.
• Commercial Retail Screeners (TradingView, Trendlyne): Display point-in-time charts for humans, but their programmatic backtest engines apply corporate revisions backwards across historical timelines.
• Over-Architected AI Platforms: Attempt to feed raw financial news into black-box LLMs without validating whether sentiment scores have any predictive Spearman rank correlation under transaction slippage.

---

4. Architecture

Sentinel enforces strict temporal boundaries by isolating data ingestion, point-in-time feature extraction, model inference, and audit verification into deterministic modules:

        ┌────────────────────────────────────────────────────────┐
        │            Raw Exchange Feeds (NSE / BSE / SEC)        │
        └───────────────────────────┬────────────────────────────┘
                                    │
                       [Ingestion & Timestamping]
                                    │
        ┌───────────────────────────▼────────────────────────────┐
        │       Immutable Append-Only Temporal Storage           │
        │   (Effective Date, Knowledge Timestamp, Supersedes ID) │
        └───────────────────────────┬────────────────────────────┘
                                    │
                                    ▼
        ┌────────────────────────────────────────────────────────┐
        │               Point-in-Time Access Gateway             │
        │       Enforces Rule: Knowledge Timestamp <= As-Of      │
        └───────────────────────────┬────────────────────────────┘
                                    │
                                    ▼
        ┌────────────────────────────────────────────────────────┐
        │            Adversarial Leakage Audit Suite             │
        │    [Label Shuffling] [Dead Features] [Future Leaks]    │
        └───────────────────────────┬────────────────────────────┘
                                    │ (Passed All Leak Gates)
                                    ▼
        ┌────────────────────────────────────────────────────────┐
        │         LightGBM Cross-Sectional Ranking Engine        │
        │   (Information Coefficient & Quintile Spread Scoring)  │
        └───────────────────────────┬────────────────────────────┘
                                    │
                                    ▼
        ┌────────────────────────────────────────────────────────┐
        │            Immutable Live Prediction Ledger            │
        │       (Cryptographically Logged Soak Window Record)    │
        └────────────────────────────────────────────────────────┘

The system is defined by three strict boundaries:

1. The Temporal Storage Layer: Every fundamental metric, shareholding pattern, and price point contains two distinct timestamps: the Effective Date (period ending) and the Knowledge Timestamp (exact millisecond made publicly available).
2. The Point-in-Time Gateway: Feature pipelines cannot query the database directly. They pass through a gateway function: get_features(ticker, as_of_date). Any record with knowledge_timestamp > as_of_date is mathematically erased from the query scope.
3. The Adversarial Test Suite: Before a model checkpoint is accepted, it must survive synthetic sabotage (feature time-shifting, label permutation) to prove it is not exploiting data leakage.

---

5. Technical Decisions

Earned Complexity: SQLite First

Rather than spinning up multi-node Cassandra or Snowflake clusters, Sentinel deliberately runs on a single-node, optimized SQLite database with WAL (Write-Ahead Logging) mode. SQLite supports atomic transactions, instant file backups, zero DevOps latency, and executes millions of indexed b-tree range queries in microseconds. The architectural motto is simple: Has reality earned this complexity? If a dataset fits on NVMe disk, distributed databases are a distraction.

Information Coefficient over Naive Directional Accuracy

Directional classification accuracy (up/down) is useless in market research because a model can be 60% accurate while losing massive capital on tail-risk events. Sentinel evaluates strategies via:

• Information Coefficient (IC): Spearman rank correlation between model predictions and forward returns.
• Information Ratio (IR): Consistency of the IC over time ($IC / \sigma_{IC}$).
• Quintile Spread: The monotonically increasing return delta between the top 20% ranked equities and the bottom 20%.

Pessimistic Cost Model

Every simulation automatically factors in Indian equity transaction realities: STT (Securities Transaction Tax), exchange turnover fees, SEBI charges, GST, stamp duty, and a dynamic slippage model scaling with median daily volume.

---

6. AI/ML Components

• Cross-Sectional Gradient Boosted Ranking: LightGBM rankers trained with pairwise and lambdarank objectives to order equities by relative strength rather than predicting arbitrary raw price targets.
• FinBERT Sentiment Parsing: FinBERT transformer models fine-tuned on Indian corporate filings and conference call transcripts, mapped through the point-in-time gateway to prevent news publication leakage.
• Permutation Feature Importance: Evaluates feature decay over multi-year regimes to eliminate ephemeral market regime artifacts.

---

7. Infrastructure

• Language: Python 3.11 with typed dataclasses and polars/pandas vectorized matrix pipelines.
• Storage: SQLite 3 with Write-Ahead Logging (WAL) and memory-mapped I/O (PRAGMA mmap_size).
• Compute: Local dedicated hardware with vectorized SIMD execution for lightning-fast backtest passes.
• Audit Trail: Immutable JSONL logs signed with cryptographic SHA-256 hashes for every out-of-sample forward prediction.

---

8. Performance

• Temporal Query Throughput: Evaluates 10 years of cross-sectional point-in-time fundamentals for 200 liquid banking equities in under 12 seconds.
• Zero Leakage Invariants: 100% of adversarial penetration tests pass continuously in automated CI.
• Evaluation Accuracy: Eliminates phantom backtest profits; models that claimed 80% annual return in conventional backtesters drop to an honest, unvarnished 14% after realistic friction and point-in-time gating.

---

9. Challenges & Realities

Indian Exchange Corporate Filing Inconsistencies

Filings on BSE/NSE historically lacked standardized XBRL machine-readable timestamps.

• Solution: Built an automated scraping and ingestion parser that cross-references exchange RSS feeds with timestamped regulatory upload headers, recording the most conservative knowable time.

Resisting the Urge to Over-Engineer

It is tempting to add multi-agent reasoning, real-time WebSockets, and deep neural networks on day one.

• Solution: Enforced Phase 0 soak window rules: until simple gradient boosted models prove an edge out-of-sample, zero complex deep architectures are permitted.

---

10. What I Learned

The Rule of Thumb: In quantitative engineering, clean data lineage beats fancy architectures every single time. A simple linear model on point-in-time data will outperform a complex deep neural network trained on leaked data in production.

Sentinel taught me deep intellectual discipline: validating software not by how impressive its metrics look, but by how rigorously it attacks its own assumptions.

---

11. Results & Current Status

• Phase 0 Operational: Successfully completed architecture and data pipeline validation across Indian banking equities.
• Live Observation Window: Operating in a continuous 30-day live out-of-sample soak window, immutably logging predictions before market open to evaluate real forward Information Coefficients.
• Engineering Foundation: Established an incorruptible quantitative research standard for future algorithmic systems.

---

Explore related articles on algorithmic determinism and systems engineering in the blog.