Incremental indexing
When you re-crawl a domain, ZENIA uses incremental indexing to avoid re-processing the entire site. Only pages whose content appears to have changed are fetched and processed again, which helps reduce unnecessary work.
How it works
HEAD-first change detection
For pages that were indexed previously, ZENIA can send a lightweight HEAD request first. If the server's
ETag or Last-Modified header matches the stored value, the page can be skipped without downloading the full content. When metadata indicates a change, or when useful metadata is missing, ZENIA fetches the full page and checks the content directly.Change detection
Content hashing
When a page is downloaded, ZENIA extracts the visible text content and generates a SHA-256 hash. If the hash matches the previously stored value, the page is marked as unchanged even if the raw HTML differs. This helps avoid unnecessary re-processing caused by small markup, script, tracking, or whitespace changes.
Statuses
Page statuses
After an incremental crawl, each page can receive one of the following statuses:
| Status | Meaning |
|---|---|
| New | The page was discovered and indexed for the first time |
| Changed | The page content changed and the page was processed again |
| Unchanged | The page was checked and no meaningful content change was detected |
| Skipped by HEAD | Server metadata such as ETag or Last-Modified indicated that the page had not changed |
| Stale | The page was not found during link discovery and may have been removed from the site |
| Failed | The page returned an error or could not be processed as supported page content |
Stale content
Stale page detection
If a previously indexed page is no longer found during link discovery, it can be marked as stale. Stale pages are not necessarily deleted immediately. Their existing crawl data may be preserved so the page can be restored if it appears again in a future crawl.
Skipped pages
Skipping unnecessary work
When an incremental crawl finishes and zero pages were added or changed, ZENIA can skip unnecessary follow-up processing. This keeps re-crawls faster for sites that update infrequently and makes it easier to see when no meaningful content changed.
First crawl
First crawl vs. re-crawl
The first crawl of a domain needs to discover and process the initial set of pages. After that first crawl, later crawls can use the incremental strategy automatically, so unchanged pages do not need to be processed again. Discovery may also take supported site access rules such as
robots.txt into account.