Status

Rate Scanner — Project Status

Last Updated: 2026-06-18 (Round 6)

---

Current Phase: Phase B — Coverage Upgrade Complete

The Rate Intelligence Engine is live and running. Daily pipeline now scrapes 14 sources, covering 29 institutions and 74 products. NerdWallet Canada replaces stale highinterestsavings.ca as primary comparison aggregator.

---

Completed Phases

Phase 1 — Foundation

• YAML manifest (rates.yaml) — institutions + products + rates
• Pydantic models with validation
• SQLite schema — insert-only rates, change detection
• CLI: rate-scanner list, rate-scanner run, rate-scanner export
• BoC Valet API scraper — prime rate + overnight target

Phase 2 — Scrapers + Change Detection

• highinterestsavings.ca aggregator scraper (16 products, flagged needs_verification)
• TD Bank scraper (Playwright) — 0.010% / 0.450% confirmed
• EQ Bank scraper (curl) — 10 products, 2.35%–3.65%
• Change detection (diff.py) — logs when rates move

Phase 3 — Export + Widget

• rate-scanner export → data/rates.json
• Benefit Snapshot widget (benefit-snapshots/) — self-contained HTML/JS/CSS
• Deployed to https://benefit-snapshots.pages.dev

Phase 4 — Big 6 Bank Scrapers

• RBC scraper (curl_cffi) — 0.550%
• BMO scraper (curl_cffi) — 0.010%
• Scotiabank scraper (curl_cffi) — 0.400%
• Tangerine scraper (curl_cffi) — 0.700%
• CIBC — Akamai-blocked; needs_verification: true, accepted as reference_only
• reference_only: true flag — Big 6 + Tangerine tagged; excluded from recommendations, used only for inertia baseline
• Widget updated — recommendation candidates filter out reference_only products
• Daily cron — 0 10 * * * = 06:00 ET; scripts/daily-run.sh → scrape → export → build → deploy
• Committed through 794775e

Phase B — Coverage Upgrade (2026-06-18)

• Root cause identified: highinterestsavings.ca severely stale (EQ shown 1.00% vs 2.75%; Neo shown 2.25% vs 3.00%)
• Neo Financial direct scraper — 3.00% (Savings), 1.25% (HISA) confirmed
• MAXA Financial direct scraper — 1.80% HISA + 5 GIC terms
• Outlook Financial direct scraper — 1.80% HISA + 2 GIC terms
• Steinbach Credit Union direct scraper — 2.00% tiered HISA
• Oaken Financial Playwright scraper — domain parked (oakenfinancial.com→/lander); covered via NerdWallet CA
• NerdWallet Canada comparison aggregator scraper — 17 institutions matched
• 6 new institutions added: Steinbach CU, Meridian CU, PC Financial, Alterna Bank, ATB Financial + new product tier for Neo
• 21 new products added (Manitoba CU GICs, credit unions, challenger banks)
• rates.yaml: Neo fixed 2.25% → 3.00%; schema_version 1 → 2
• Pipeline: 8 scrapers → 14; 74 products (was ~53)
• Real run confirmed: 35 new rate rows written (2026-06-18)

---

Phase C — Dashboard UI Overhaul + Debt Rate Expansion (2026-06-18)

• Dashboard UI overhauled (rates.astro): H1 centered “myBetterRates Summary”, section banners with inverted backgrounds (Savings=emerald, Debt=amber), no calculator inputs, simplified columns
• Savings Accounts: now shows only best chequable (EQ 2.75%) + best HISA (Neo 3.00%), sorted highest first
• GICs: deduplicated to best rate per standard term (5 rows: 6mo/1yr/2yr/3yr/5yr), sorted highest rate first
• All debt tables sorted ascending (lowest rate first)
• Summary chips corrected: “Best low-rate card” now shows lowest rate (8.99% MBNA True Line Gold)
• Real mortgage data added: 5 products (3.35% variable → 4.69% 1yr fixed), sourced from RateHub 2026-06-18
• broker_network institution added for broker-channel mortgage rates
• Auto Loans section added: New vehicle 7.20%, Used 8.49% (Big 6 starting rate, RateHub 2026-06-18)
• Mortgages now show results (not “coming soon” / competitor links)
• rates.yaml: 74 → 76 products, 56 verified; broker institution_type added
• Deployed to https://mbr-site.pages.dev/rates — all checks pass

Phase D — Mortgage & Auto Loan Expansion (2026-06-18)

• ratehub_mortgages.py scraper built: curl_cffi (no Playwright), parses RateHub table — 9 products daily
• ratehub_car_loans.py scraper built: parses RateHub lender table — Dealerhop 6.99%, Big 6 7.20%
• dealerhop institution added to manifest (best auto loan rate in Canada: 6.99%)
• New mortgage products added: 2yr, 4yr, 7yr, 10yr fixed + 3yr variable — manifest: 82 products (was 76)
• 3yr fixed mortgage corrected: was 3.84% (mislabeled from Round 5 — it was the variable rate). Fixed → 4.04%
• 3yr variable mortgage added: 3.84% (Prime − 0.61%) — separate product in manifest
• mortgage_best_5yr_fixed_uninsured updated to 4.24% (estimated), needs_verification: true
• Pipeline: 14 scrapers → 16 scrapers; 9 new mortgage rows + 2 auto loan rows written today
• Dashboard updated: Mortgages now shows Term + Type columns; 9 rows; variable mortgages clearly labelled
• Dashboard auto loans: Dealerhop 6.99% ranks #1 (lowest); Big 6 7.20% #2; 8.49% used vehicle #3
• Deployed to https://mbr-site.pages.dev/rates — all checks pass

---

Active Work

None — Phase D mortgage/auto loan expansion complete. Daily pipeline running autonomously.

Key metrics (2026-06-18 Round 6):

• 16 scrapers in daily pipeline
• 82 products in manifest (31 institutions)
• 61 verified products exported
• Inertia baseline: 0.010% (TD Bank)
• Best HISA: 3.00% (Neo Financial) → delta $2,990/yr per $100k
• Best GIC: 3.85% 5-year (MAXA Financial)
• Best mortgage: 3.35% 5yr variable insured (Broker Channel, scraped daily)
• Best auto loan: 6.99% new vehicle (Dealerhop)

---

Next (Phase E)

In priority order:

1. Hassle-engine scaffold — core algorithm: read rate_changes from DB, evaluate user threshold (e.g. “alert me if delta > 0.5%”), return alert payload. No delivery yet.
2. needs_verification products — 7 aggregator-sourced products need manual confirmation: Hubert 2.30%, Bridgewater 2.70%, Peoples Trust 1.20%, PC Financial 2.20%, Alterna Bank 1.05%, ATB Financial 1.80%, Meridian CU 0.85%
3. Mortgage uninsured rates — build RateHub uninsured-specific scraper to confirm mortgage_best_5yr_fixed_uninsured
4. Oaken Financial — domain parked at oakenfinancial.com; confirmed live at oaken.com (2.80% savings rate)
5. Cancer50Pledge.ca Phase B — custom domain, pledge form with email capture, analytics
6. Benefit Snapshot custom domain — snapshot.mybetterrates.com or rate.mybetterrates.com
7. MBR consumer site — the actual mybetterrates.com web app (SvelteKit)
8. National Bank scraper — National Bank of Canada (last of Big 6 not yet scraped)

---

Architecture Decisions (Locked)

Decision	Choice	Reason
Language	Python + uv	Clean deps, fast install, matches GlobalDevRules
Storage	SQLite (insert-only)	Audit trail, no deletes, perfect for rate history
Scraping	curl_cffi first, Playwright fallback	curl_cffi beats Playwright on TLS fingerprinting for most banks
Deploy	Cloudflare Pages	Free tier, edge CDN, wrangler CLI
reference_only flag	In YAML manifest only (not DB)	Static property — doesn’t change over time
Inertia baseline	TD Bank 0.010% (Big 6 worst)	Confirmed systemic: BMO also 0.010%

---

Feedback

(leave blank — AI acts on feedback here before proceeding)