Market Data Retention Specification
Purpose
Defines the capture schedule and retention policy for TBA pricing data from dataQollab, ensuring historical market context is available for hedge analysis, trade decision auditing, SFAS/ASC 815 compliance, and retrospective performance analysis.
Capture Schedule
Scheduled Snapshots (10 per business day)
9 hourly snapshots during market hours plus the SIFMA close mark:
| Time (ET) | Snapshot | Significance |
|---|---|---|
| 8:00 AM | Hourly #1 | Pre-open mark — sets morning rate sheets |
| 9:00 AM | Hourly #2 | |
| 10:00 AM | Hourly #3 | |
| 11:00 AM | Hourly #4 | |
| 12:00 PM | Hourly #5 | Mid-day mark — mid-day repricing reference |
| 1:00 PM | Hourly #6 | |
| 2:00 PM | Hourly #7 | |
| 3:00 PM | Hourly #8 | 3:00 PM mark — used by some shops as pricing benchmark; dataQollab provides price_mark_at_3pm |
| 4:00 PM | Hourly #9 | |
| 4:45 PM | Close mark | SIFMA recommended close of TBA trading — the industry standard end-of-day mark for P&L, hedge accounting, and overnight risk positions |
Event-Driven Snapshots
In addition to the scheduled captures, take a snapshot at the moment of every trade execution on PSX. When WTPO confirms a loan purchase, the market prices at that instant are captured and linked to the trade record.
This provides "what was the market when we made this decision?" context without relying on interpolation between hourly marks.
Non-Business Days
No snapshots on weekends or market holidays (SIFMA holiday schedule). The market doesn't move when it's closed.
Data Per Snapshot
Each snapshot captures the full dataQollab price grid:
- 403 rows (13 instrument lines × 31 coupon steps)
- Fields per row: agency, coupon, delivery month (×4), bid/ask/mid, deltas, settlement dates
| Field | Source |
|---|---|
snapshot_timestamp | When the capture was taken |
snapshot_type | scheduled or execution |
execution_reference | Trade ID (for execution snapshots, null for scheduled) |
mbs | MBS instrument identifier |
settlement_month | Delivery month |
bid_price | Bid price |
ask_price | Ask price |
mid_price | Mid-market price |
delta_vs_3pm | Change from 3:00 PM mark |
delta_vs_445pm | Change from 4:45 PM mark |
price_mark_at_3pm | Price at 3:00 PM |
price_mark_at_445pm | Price at 4:45 PM |
settlement_date | Settlement date |
Storage Estimates
| Metric | Value |
|---|---|
| Rows per scheduled snapshot | 403 |
| Scheduled snapshots per day | 10 |
| Rows per day (scheduled) | 4,030 |
| Estimated execution snapshots per day | 1-3 (WTPO volume) |
| Rows per day (execution) | ~1,200 (3 × 403) |
| Total rows per day | ~5,200 |
| Business days per year | ~250 |
| Rows per year | ~1,300,000 |
| Bytes per row (estimated) | ~200 |
| Storage per year | ~260 MB |
| 3-year retention | ~3.9M rows / ~780 MB |
Storage is trivial. Store everything. Disk is cheaper than regret.
Retention Policy
| Data | Retention Period | Reason |
|---|---|---|
| All scheduled snapshots | 3 years minimum | Audit retention period for hedge accounting (SFAS 133 / ASC 815) |
| Execution snapshots | 3 years minimum | Trade decision audit trail |
| End-of-day (4:45 PM) marks | 7 years (consider) | Some regulatory retention requirements extend beyond 3 years |
Data is append-only — historical snapshots are never modified or deleted within the retention window. Purging beyond the retention period is optional (storage is cheap enough to keep indefinitely).
Use Cases for Historical Data
1. Hedge Effectiveness Testing (SFAS 133 / ASC 815)
Auditors require proof that hedges were effective during the reporting period. End-of-day marks for every business day, compared against the loan pipeline's value changes over the same period. The 4:45 PM close mark is the standard reference.
2. Trade Decision Auditing
"Why did we buy these loans at this price?" — compare the execution snapshot (market at trade time) against the principal's offer price to verify the spread was positive and within policy. Regulators and internal compliance may ask this question months later.
3. Rate Sheet Accuracy Validation
Compare the TBA prices used to generate WTPO's rate sheet against the actual market at the time sellers received the sheet. If the market moved 0.5 points between rate sheet generation (8 AM) and seller acceptance (2 PM), the hourly snapshots show exactly how much drift occurred.
4. Performance Attribution
"Did we make more money in Q1 or Q2, and why?" — historical TBA prices combined with historical trade data enable period-over-period analysis of trading performance, separated from market movements.
5. Market Volatility Analysis
Hourly snapshots reveal intraday volatility patterns. "How much does the market typically move between 8 AM and 4:45 PM?" — useful for setting rate sheet validity windows and repricing triggers.
Implementation Notes
API Call Pattern
Each scheduled snapshot calls https://api.mbsmkt.com/p/prices/snapshot with PowerSeller's API key. The response is parsed and inserted into the market_pricing_history table with the snapshot timestamp and type.
Scheduling
Use a cron job or PSX's worker queue (arq) to fire at the 10 scheduled times. The 4:45 PM snapshot fires at a non-hourly time — handle explicitly.
Execution snapshots are triggered synchronously when a trade is confirmed — the trade confirmation handler calls the snapshot API before returning.
Data Source
All snapshots come from the same dataQollab API endpoint. No separate vendor relationship. No additional cost. See dataQollab TBA Pricing Discovery.
Where This Data Lives
Per the PSX SA's architecture answers: dedicated market_pricing table in PSX Postgres. Historical snapshots append to market_pricing_history. Redis cache holds the latest snapshot for hot-path access during rate sheet generation.
This data is NOT replicated to PSSaaS. PSSaaS accesses pricing via PSX's ephemeral pricing API when needed (per ADR-042 / PSX-to-SaaS integration).