# Bellingham Middle Housing Tracker

Personal research tool for Ted (a builder in Bellingham, WA). **Not for reporting or publication** — the goal is maximum information for Ted's own build decisions, so messy/inferred data is fine as long as confidence is labeled.

## Why this exists (read first)
Ted has already studied Bellingham's middle-housing ordinances (Infill Toolkit "ITK", HB 1110, ADU rules) to figure out *what he's allowed* to build. This project answers the next questions:
1. **Market:** what finished middle-housing products (townhouse, cottage, small lot, duplex–sixplex, ADU…) actually sell or rent for, and how fast.
2. **Rent vs. sell:** which strategy builders are choosing per product type, and what the numbers look like for each.
3. **Rubber-hits-road check:** find real finished projects the City actually approved and finaled, by type, so Ted can study the precedents (permit trail, timelines, who built them).

This will be a major, ongoing area for Ted, worked on over many sessions. **Keep this folder's context clean:** update `docs/` as findings change, don't leave knowledge only in chat.

## Deliverable shape (Ted's preference)
- A **local browser app opened via `file:///`** (not a claude.ai Artifact). Ted loads `index.html` directly.
- So: no fetch() of local JSON (blocked under file://). Python refresh scripts write `data/*.js` files (`window.MH_DATA = {...}`) loaded with `<script src>`. Same pattern as Ted's "standard house index" dashboard (`~/Documents/Projects/Claude Projects Private/Housing Book/standard-house-index/ui/`) — it's the visual bar to aim for (photo cards, map, rich detail). Ted loved that one.
- CDN libs (e.g. Leaflet) are OK, or vendor them into `ui/lib/` like the index project did.
- User notes/overrides in localStorage with JSON export/import.

## Scope decisions (2026-10-29)
ADUs are integrated as one more product type, not a separate tab. Geography is Bellingham + Ferndale + Blaine (+ Lynden). Refresh weekly or less, on demand. Details are in `docs/roadmap.md` → Decisions.

## Folder map
- **`pipeline/`: the real pipeline (2026-10-29).** Run `zsh pipeline/refresh.sh`. Steps:
  1. `pull_permits.py` → `data/raw/permits.json`
  2. `county_parcels.py` → `data/raw/county_parcels.json`
  3. `rental_registry.py` → `data/raw/rental_*.json`
  4. `enrich_assessor.py` caches assessor pages in `cache/assessor/`, sequential, resumable, stops itself on repeated errors; a full first run is about 2.5 hrs.
  5. `build.py` → `ui/data/data.js`

  `universe.py` decides which parcels count as middle housing. Grouping, status inference and the evidence strings live in `build.py`; read its docstrings before changing the rules.
- **`ui/index.html`: the app.** UI model Ted settled on (2026-10-30): a **mode switch** [All projects | On the market] chooses WHAT you're looking at; the header count and right pane follow the mode (project summary vs. listing stats). **Views** [Map | List | Stats] choose HOW, and never change the right pane. List = project cards in all-projects mode, the listings table in market mode. Stats is a full-width report (no right pane). Header filters narrow either mode. The right pane shows the clicked project's detail in any mode or view. The rest of this entry: Open it via file:///. Leaflet is vendored in `ui/lib/`. Notes and the lookup queue live in localStorage (mh.notes / mh.queue), with Export/Import buttons. To test it in Claude's browser pane, use the `ui` config in `../.claude/launch.json` (python http.server on :8790); the pane can't load file:// with relative scripts.
- `data/lookups/*.json` (optional): dated snapshots from on-demand Zillow / property-manager lookups. `build.py` merges them by prop_id or address: `{"prop_id"|"address", "date", "source", "url", "events":[{"date","kind","price"}], "notes"}`.
- `data/redfin_search/<date>.json`: raw Redfin search snapshots (they accumulate, so history builds up). `pipeline/redfin_merge.py` → `data/redfin.json` (keyed by prop_id) → build.py. Statuses `for_sale` and `sold_recent` come from it, and `photos[0]` becomes the card image.
- `pull.py`, `mm.py`, `group.py` — **original** (2025+ only) pipeline → `middle_housing_permits.csv`, `middle_housing_projects.csv`. Superseded in scope by `feasibility/pull_all_mm.py` (2015+, more housing types). Intermediate JSONs they reference are not saved.
- `etrakit_scrape.js`, `analyze_etrakit.py`, `etrakit/` — scraper for permits.cob.org eTRAKiT (needs Ted logged in, run in browser) + per-project timelines for a hand-picked finaled subset.
- `feasibility/` — 2026-10-29 data-source feasibility pass: `pull_all_mm.py`, `assessor_probe.py`, `redfin_probe.js`, `raw/`, `cache/assessor/`.
- `docs/feasibility-2026-10-29.md` — **what each data source gives, hit rates, examples, gotchas.** Read before touching data acquisition.
- `docs/roadmap.md` — proposed architecture, status model, next steps, open questions.
- `data/manual/` — files Ted downloads or enters by hand (e.g. his own duplex rent history, pasted Zillow links).

## Data-source summary (details in docs/feasibility-2026-10-29.md)
| Source | Access | Gives |
|---|---|---|
| City permits ArcGIS (`maps.cob.org/arcgis3/.../Maps/Permits/MapServer/0`) | open REST | every permit since ≤2015, Housing_Type, units, sqft, dates, APN, lat/lon |
| City tax parcels (`.../Maps/Grp_Land/MapServer/5`) | open REST | owner, owner mailing addr, Property_ID, subdivision, land use |
| Whatcom County assessor (`property.whatcomcounty.us/propertyaccess/Property.aspx?prop_id=`) | plain HTML, cache it | owner, mailing addr, value, building type+sqft, **full sales history with prices & grantor/grantee** |
| Redfin search pages (built-in browser) | ~25 page loads per refresh; `pipeline/redfin_search.js` + `redfin_merge.py` | sold-last-year + for-sale-now for all 2015+ homes: price, sold date, beds/baths, sqft, days listed, photo, description | MLS list/pending/sold history, some rental listings, photos, descriptions |
| Zillow | single page views OK in browser; **bulk fetch → 403 bot block** | best rental history (Zillow Rentals list/remove dates + rent), Rent Zestimate |
| Whatcom County parcels (`wcgis.whatcomcounty.us/.../WhatcomCo_Property/MapServer/1`) | open REST | **county-wide**: use code, yr_blt, sqft, owner, values. Universe for Ferndale/Blaine/Lynden (no permit data there) |
| NWMLS (via Ted's agent) | CSV exports → `data/manual/nwmls/` | sold/leased comps; spec in `docs/nwmls-request.md` |
| City rental registration (public ArcGIS Online layer `Plan_RentalRegistrations`, see feasibility doc §5) | open REST | 5,386 registered rentals by APN + 25k inspections; 45% of finaled middle-housing permits match |

## Guardrails learned
- Before scraping a page, look for the official data behind it: ArcGIS dashboards have public FeatureServer layers (found via `arcgis.com/sharing/rest/content/items/<id>/data`). That is how the rental registry was found.
- Do **not** spoof headers or retry around bot blocks (Zillow 403, cob.org PDF 403). That was flagged and denied on 2026-10-29. Use the built-in browser for normal page visits at human pace, or ask Ted to download/visit manually.
- **Redfin: search pages yes, API no.** Recent sales and listings come from ordinary Redfin search pages (allowed); see `docs/redfin-search.md` for the routine. redfin.com/robots.txt disallows `/stingray/` (the JSON API used by `feasibility/redfin_probe.js`). A 40-address probe is one thing; a 2,000-unit pass is crawling. Decided 2026-10-29. Normal search pages and property pages are fine at human pace. Also: the built-in browser blocks a redfin.com tab from reaching localhost (fetch and iframe both get ERR_BLOCKED_BY_CLIENT) and from writing the clipboard, so there's no easy way to hand bulk data out anyway.
- Assessor site returned 503s after ~650 pages with 3 parallel workers (2026-10-29). Fetch **one at a time**; the script stops itself after 3 errors in a row. Resume later, never hammer.
- Be gentle: ~1 req/sec to assessor, cache everything under `feasibility/cache/` (or later `cache/`).