Scraping Adversarial Municipal Portals

The Challenge

Most California cities publish their building permits, and almost none of them publish the same way. The San Francisco Peninsula alone runs six vendor portal products (eTRAKiT, on at least two incompatible forks; Tyler Technologies' EnerGov and Civic Self Service; Accela; Clariti; PermitStack), one municipal open-data feed, and one city whose only complete record is a monthly report it posts as a document. Searches are login-gated, JavaScript-rendered, and driven by ASP.NET postbacks that rewrite the form under you. Two of the portals answer only to a real browser.

A blocked request is loud: it throws, the run goes red, somebody fixes it. The failures that cost you are the ones that return a well-formed answer to a question you did not ask.

Failure one: the empty list that looks like a quiet month.

One city's portal login was being rejected on every run. The scraper printed a warning, returned an empty permit list, and the pipeline reported success. A city we actively cover disappeared from the product for weeks and nothing alarmed.

Then the absence turned into a zero. The city's demolition-permit count rendered as 0, indistinguishable from a city that had filed none, and that zero reached public copy written for builders in that market.

Failure two: the result set that is not the answer.

Every eTRAKiT host caps a search result. It applies the cap before sorting, it emits no error, and once the grid's page size is raised past the cap the pager that would have read "page 1 of N" disappears. Nothing on the page distinguishes a truncated result set from a complete one except its size.

The rows it keeps follow the portal's own internal order, which skews heavily oldest-first, so the truncation drops the newest permits: the only ones a recent-activity scrape exists to find.

What We Built

AddressIntel's permit layer is a fleet of nine scrapers, running Selenium and undetected-chromedriver on a self-hosted runner, over six vendor portal platforms, one municipal open-data feed, and one city's published monthly reports, plus a records-request ingest for the towns that cannot be scraped at all. As of 2026-09-16 it holds 187,838 permits across 20 permitting jurisdictions: nineteen cities and towns, plus unincorporated San Mateo County, whose rows carry the names of seven coastal communities. 177,522 of the permits carry an assessor's parcel number and 128,324 carry a declared valuation above zero.

Two things about that count belong next to it. Coverage is uneven, from 53,160 rows in Palo Alto to 28 in Foster City, and a city with few rows is either quiet or under-collected, which the rest of this piece is about telling apart. And 39,830 rows, 21% of the total, carry no issued date: 37,235 of those are Palo Alto's, from an open-data export that publishes an application date and a lifecycle status for each permit (33,641 of them Finaled) but no issue date. The remaining 2,595 are spread across five other sources with mixed statuses, including 372 marked Issued that carry no date for it. One Saratoga row is dated 2026-10-03, seventeen days after this was written, with an application date of 2025-10-01. It is stored as the portal printed it.

1. Measure the cap; never assume it

Because nothing on the page announces truncation, the size of the result set is the only tell available, and it has to be read before anything is trusted. Each host's cap was measured directly on 2026-08-25, one query per host, page size raised to 2000 so the whole set arrived at once:

The cap ranges from 100 to 1000 across four hosts of nominally the same product, so it cannot be hardcoded once and forgotten. It is also independent of page size: page one of the un-resized grid reports "page 1 of N" with N times the page size equal to the cap exactly, and raising the page size past the cap returns the cap with no pager at all.

Sorting does not help. Sorting AT LEAST 01/01/2026 on the Hillsborough host by issued date descending, which is what the scraper's own sort helper does, still returns that query's 500 rows stopping at 2026-04-22, while the city has been issuing permits continuously since. (That is a different query from the one in the table above, which is why its cut-off date differs; both saturate at 500.) Sorting reorders the survivors. It does not change which rows survived.

Two hosts in the fleet are login-gated, so their caps are unmeasured rather than absent, and the code keeps that distinction. For those, the fallback test is a total landing exactly on a round number: a result set is a count of permits a city happened to issue, and that count landing on exactly 500 by coincidence is far less likely than it landing there because something clipped it.

def etrakit_is_saturated(total_rows, url=None, cap=None):
    """True when a result set of `total_rows` is sitting on its host's cap.

    Saturated means TRUNCATED: the portal returned as many rows as it will
    ever return, so there are almost certainly more records matching the
    search than came back, and the ones missing are the newest.

    Deliberately >= rather than ==: a host that raises its cap, or one whose
    cap was measured low, must still trip this.
    """
    if total_rows is None:
        return False
    cap = cap or etrakit_result_cap(url)
    if cap:
        return total_rows >= cap
    return total_rows in ETRAKIT_ROUND_CAPS

2. Find the one bucket the platform cannot truncate

Detecting truncation is only useful if there is something better to do than report it. The standard answer, bisect the query until each half fits, is unavailable here: the search offers one criterion and five operators (BEGINS WITH, CONTAINS, EQUALS, AT LEAST, AT MOST), there is no BETWEEN and no second field, the string operators reject a partial date outright, and AT MOST truncates from the same end, so an AT MOST query on the Atherton host comes back with 100 rows whose newest is 2008-04-28.

That leaves EQUALS on a single calendar day as the only bounded bucket the platform offers. Measured across the four date-search hosts, a single day returns 5 to 22 rows against caps of 100 to 1000. A bucket that small never reaches the cap, and the scraper checks each day's total anyway, so a day that did saturate would be reported rather than half-read.

The walk runs newest-first on a 45-day budget, because each day costs a postback and its settle time, and the freshest permits are the ones worth spending the budget on. Whatever the budget does not reach is named in the run log and recorded as a source failure. A historical backfill asks for 1825 days, which this path will not walk; the alternative is a backfill that returns the portal's oldest 500 rows and calls itself complete.

What the instrument reads. Across the sixteen scheduled runs from 2026-09-10 to 2026-09-16, twelve reached the Atherton host's date search. All twelve saturated its 100-row cap and fell back to the day-walk; the other four were cut off by the time budget before reaching it. None of the sixteen recorded an uncovered window, so the fallback never had to report a gap it could not close.

The outcome is visible in the database, which holds 138 Atherton permits issued on or after 2026-08-16, for a search the portal will not return more than 100 rows of. The town's 1,298-page records ingest cannot account for the surplus: it landed on 2026-08-05, before that window opens. Saturation on this host is the steady state, and without the fallback every one of those twelve runs would have reported success while handing back the oldest hundred rows of the window.

The budget that binds is max_permits, at 50. Every one of those runs stopped after 12 to 16 of the 31 days on that cap, which the walk inherits from the plain search. The 45-day figure is the ceiling on how far back a walk may reach.

A saturation the day-walk fully recovered from costs time and is therefore not recorded as a failure. Only days the walk could not cover are. A busy city would otherwise red the pipeline twice a day forever, which is the alarm-nobody-reads failure mode the health gate exists to prevent.

3. Two gates, and the incident between them

The health registry records only failures the scrapers know about: a rejected login, an unreachable portal, a saturated search that could not be recovered. It does not infer breakage from a low permit count, because a quiet city and a broken scraper both produce zero permits, and guessing between them is how you build an alarm that cries wolf and then gets ignored. If a scraper cannot tell you it failed, that is a gap in the scraper.

That rule is right for the registry, and it was not enough for the pipeline. On 2026-09-01, two cities were found silently dead in production: Sunnyvale had added no permit row for 14 days, Woodside for 33. Every run in that window reported success, and three separate guards missed it. The registry saw nothing, because neither city failed in a way its scraper knew about: Woodside's date-walk searched a portal that has no date search and got a legitimate empty result, and Sunnyvale's aggregator answered HTTP 200 with frozen data. The freshness check read MAX(scraped_at) across every city at once, which San Jose alone holds at today forever, so it reported all sources fresh on the morning both cities were 14 and 33 days dead. The third guard reported which targets the time budget had cut off; Sunnyvale had been reached in 8 of the previous 11 runs and ingested nothing on every one.

A permit source can fail by succeeding at nothing, and nothing was watching whether rows actually landed, per city, over time. A second gate now does that. It infers from volume, the inference the registry refuses to make, and it makes it in one place, explicitly, with a threshold that can be read.

Sizing that threshold is where the interesting mistake lives. The obvious approach is to learn each city's normal silence from its own history, and that was tried first. Measured against the production snapshot, a dozen unrelated cities shared an identical 13-day gap in August 2026, a rotation failure that had reached only half the target list for twelve days. A threshold learned from that history would have set Sunnyvale's tolerance to 20 days and slept straight through its 14-day outage. A guard that learns "normal" from a record containing its own failure mode gets blinder every time it fails. The threshold is sized from how much each city normally files instead, with a ten-day floor.

Three details of the registry follow from the same reasoning.

• Failures flush on every failure. The scrape executes under a timeout that terminates it when the budget expires, routinely. Anything saved only at the end of the run is lost exactly when a slow, failing source is the likeliest cause.
• The gate runs last, after the sync and the data-store save. The run goes red, but every byte the scrape did manage to collect is already persisted. A red run that kept its data is the goal.
• Known-unscrapable sources report without reddening the run. Where the fix is a new data source rather than a retry, a permanent red is a switched-off alarm. Entry into that list requires a tracked issue and a stated reason.

The retry policy is drawn on the same line. A timeout, a stale element reference, an intercepted click, a grid that did not re-render, an ASP.NET postback that never landed: these differ between two runs against unchanged markup, so they get one more attempt with a fresh driver. Two red runs in August 2026 were of this kind, one dying on an index error and one on a renderer timeout. For the first, the run ninety minutes earlier had walked the identical pages, blocks and pagers to a clean finish; that comparison was not made for the second. Neither was a markup change, and a single retry would very likely have absorbed both. In both, the sync and the data-store save had already succeeded, so they were red purely on an exit code.

NoSuchElement is excluded from that list. An element that is simply absent is the signature of a portal whose markup changed, which is real breakage and must red the run on the first attempt.

One inference across sources is safe. One quiet city is ordinary. Every city sharing a single scraper going quiet at once is a fact about the shared code, so a platform whose every source came back empty is reported as a known failure.

4. Knowing when scraping is the wrong answer

One town's portal has no issued-date search at all, only permit number and site address. A number-walk is technically possible; the town's permits carry year-prefixed series identifiers that the fleet's number-walk path already enumerates elsewhere. It is still the wrong fix, because that portal's public detail panel carries no declared valuation and no parcel number, and declared valuation is the field the town is carried for. A number-walk would restore row count while losing the column that matters, at roughly thirty-six seconds per permit. The route out is a recurring public-records request, and that channel now carries the town's entire permit record. Re-measured at publication against the September refresh, received 2026-09-11: 5,237 permits, of which 2,460 carry a declared valuation totalling $322,162,086. The town confirmed on 2026-09-01 that a standing arrangement is not available, so each refresh is its own request.

That entry in the registry also carries its own correction. It used to assert that the number-walk was impossible because the town's permit identifiers were pure numeric strings with no series prefix. They are not. The claim had been written while the town had zero scraped rows and nobody had yet seen a real permit number from it, and it survived until a records delivery revealed the actual format. Written into a code comment, an assumption made in the absence of data reads exactly like a measurement.

The results grid clips long descriptions to thirty characters server-side, with a literal ellipsis and no title attribute holding the full string, so the complete value is not on the page. The ellipsis is kept rather than stripped: it is the only record that the stored value is a prefix, and stripping it would make a clipped string indistinguishable from a complete one. Full strings arrive later from the sources that print them whole, and the writer never overwrites a value it already holds, so a clipped scrape cannot regress a complete record.

What It Shows

A permit table assembled from these sources can carry a claim about its own coverage. The saturation check, the day-walk, the two gates and the retry line each turn one specific silent failure into a logged one, and the numbers above are what they read on a given week. When a vendor changes a form control, the run goes red and the data it collected stays saved.

What It Proves to a Client

A scraper that could not report its own failure produced a zero, and that zero became a published statistic. The pipeline now records "unknown" separately from "none", and the difference is what a buyer of this data is paying for.