# Search Indexes

> Search indexes over your website, your files or a collection, searched by you, your loops, your visitors and your servers

**Reactor search** makes your content searchable: a website, files you upload, or the records
of one of your collections. It is a product of its own. You can use it with no loop, and a loop
can use it too.

Every search reads one or more **search indexes**. You create an index once, on the account, on
the **Indexes** page. Then four kinds of searcher can search it:

- **You**, on the console's **Indexes** page. See [Search your indexes yourself](#search-your-indexes-yourself).
- **Your loops.** A loop names the indexes its runs may search. See
  [Letting a Loop Search](/docs/guide/search-indexes.md).
- **Your visitors**, through a published loop's search box or its search API. See
  [Let your visitors search](/docs/guide/search-indexes.md#let-your-visitors-search).
- **Your servers**, with a search key and no loop. See [Search from your server](#search-from-your-server).

How to write a search is on [Writing a Search](/docs/search/writing-a-search.md): words, an exact
phrase, words with a typo, a pattern, and words or meaning.

## Create an index

On **Indexes**, press **New index** and choose what it is **built from**. That choice never
changes.

| Built from | What it holds |
|---|---|
| **A site crawl** | The pages of one website, fetched again daily or weekly |
| **Files you upload** | Markdown, text and HTML files, and PDFs that have a text layer |
| **A collection's records** | The fields you choose from one of your collections, kept up to date as runs write and delete records |

Every index has a **name** for people, such as "Product documentation", and an **id** that a loop
uses to name it, such as `docs`. The id is a lower-case letter, then lower-case letters, digits
or underscores. The form fills in the id from the name, for example `shop_hours` from "Shop
hours", until you type your own.

A new index is **untrusted**: what it finds counts as outside text, which limits what a loop that
searches it may also do (the [Rule of Two](/docs/guide/permissions-and-trust.md#the-rule-of-two)). An **owner** of the
account can mark it trusted on its page, with **Mark trusted…**, once it exists; a member sees the
mark and cannot change it. Mark an index trusted only for content you control. Trust also changes
what a loop that searches the index may do; see
[What an index lets a loop do](/docs/guide/search-indexes.md#what-an-index-lets-a-loop-do).

One choice applies to every kind:

- **Search by meaning as well as by words.** On by default. With it on, the index finds a
  passage that says the same thing in other words, as well as the exact words.

### A site crawl

Give the crawl an **address**, such as `https://docs.example.com/guide/`. The crawl reads that
host under that path, and nothing outside it. It reads the site's `robots.txt` and sitemap and
follows the links it finds inside that scope. Choose how often it fetches the site again,
**daily** or **weekly**, and **at most how many pages** it stores: 2,000 by default, and at most
10,000.

The first crawl usually starts within an hour of creating the index; **Fetch now** starts it at
once. Until then the list reads **First crawl soon**, never **Ready**. An index of files or of a
collection's records reads **Empty** in the list while it holds nothing, and shows on its own page
how many documents it holds and how many are still being indexed. A page that only shows its text after its
scripts run, as in a single-page app, is opened in a browser so that its text is found. The crawl
reaches public sites only, and sends no login.

A crawled page takes its **title** from the first heading of its text, so a search result shows
the page's own name and not the site's name repeated. A page with no heading keeps the title the
site gives it.

A crawled page keeps its **content** and leaves out the site around it: the navigation, the
header and footer, the sidebar, and furniture that documentation sites put beside the text, such
as an "On this page" list, a version badge, a "Was this page helpful?" box or a "Last modified"
line. A footer the site writes as ordinary text in the page, with nothing marking it as one,
stays in.

The index shows its **last crawl** in words: whether it is crawling now or finished, how many
pages it fetched, stored, rendered in a browser and failed, and what it cost. A crawl that runs
out of money stops, keeps the pages it fetched, and says so.

**Each account has a monthly crawl allowance**, set by the platform: about 180,000 pages a month,
across all of the account's crawl indexes. When it is used up, a crawl that is running stops and
keeps the pages it fetched, a new crawl does not start, and the index says **monthly crawl
allowance**. It renews on the 1st (UTC). Fetch now refuses in the same words, and the press does
not count. If your account needs more, contact us.

**Fetch now** crawls the site again at once, when you changed pages and do not want to wait for
the schedule. It starts the same crawl the schedule runs, and costs the same. An account can
press it 6 times a day (the day ends at 00:00 UTC); a press that starts nothing does not count.
It starts nothing while a crawl of that index runs, and says when that crawl started. The schedule
keeps its own crawls either way.

A crawl that finishes removes the pages it no longer reaches: a page now outside the address, a
page the site skips or refuses to crawlers, and a second address that serves the same text as a
page it kept, so a search never finds one page twice. A page that fails to load once keeps its
place. A crawl that stops early, or that reaches its page limit, removes nothing.

A crawl that reaches the end of the site but keeps **no page** also removes nothing. This happens
when the site shows a placeholder or is down. The index says **crawled: no page kept**, keeps
what earlier crawls stored, and Home names it until a crawl keeps a page again.

### Files you upload

Open the index and press **Upload a file**. The index takes Markdown, plain text and HTML files,
and PDFs with a text layer. A scanned PDF has no text layer, so the index refuses it and says
why. Each document shows its state: **pending** until it can be searched, then **indexed**. To
take a file out of the index, press **Remove…** beside it, then **Remove it**. The console asks
first because a removed file cannot be put back: upload it again to restore it.

A file added through the API can also name its **source**: the address it came from, such as the
web page it copies. A search result then shows that address, so an answer can cite where the text
came from. The address must start with `https://` or `http://`.

### A collection's records

Name the **collection** and the **fields to index**, such as `company, notes`. The index starts
with the records already stored, then follows every record a run writes or deletes.

To show records that describe the same thing as one result, name the field that identifies it
under **Same item when this field matches**, such as `sku`: two listings of one product then
show as one search result. It is a text or number field the collection declares, and not one that
holds what people typed. A record without a value in it is still its own result. You choose this
field when you create the index and cannot change it later; to fold on another field, create a
new index.

**Records can hold personal data.** With search by meaning on, their text is sent to the
embedding provider.

## Search from your server

An upload or crawl index can also be searched with no loop, by a server of yours. On the index's
page, an owner mints a **search key** under **For a server of yours: search keys**; the key searches that index and does
nothing else. Each search is charged at the published price per search. The request, the answer
and the refusals are on [Search API](/docs/api/search.md). A collection index has no search keys: its
records can hold personal data.

## Search your indexes yourself

On **Indexes**, press **Search every index** above the list to search all your indexes at once,
without opening one. An index's own **Try a search** tab opens the same search, starting on that
index. Type what you are looking for, then choose:

- **Find** — the words, the exact phrase, the words allowing typos, or a pattern (one term).
  [Writing a Search](/docs/search/writing-a-search.md) shows each with an example.
- **In** — every index, or one.
- **By** — **Both**, **Words only** or **Meaning only**. Words find an exact name, such as a
  command-line flag. Meaning finds a passage that says the same thing in other words. **Both**
  ranks the two together, as a loop's own searches do. Every search is charged a small fixed
  amount for running it. A search by meaning also embeds your query once, at the index's
  embedding price. Your wallet pays both.

Each result names its index and its title, and says when the index is untrusted. Its passage
reads as text, and code in it keeps its lines, in a monospace box of its own that ends with "…"
when the passage stops inside the code. Press a result's title to
read the document from that place.

A document is **one result**, at its best-matching section. When other sections of the same page
also matched, the result says so, for example **+2 more sections**; its title opens the whole page.
A page crawled into two of your indexes is one result too, and so are records with the same
value in their index's **Same item when this field matches** field. A run's agent gets the same
results, and is told that the rest of each page can be read.

## How results are ordered

**Search order** on **Settings** says in one line what your searches use now, such as *"Every
search uses Fuse"*. Under **More** it chooses how searches are ordered, separately for four kinds
of search: **A run's
agent**, **Your search** on Indexes, **Your visitors** on your site, and **Your search keys**
(a server of yours, through the [Search API](/docs/api/search.md)). Each one is set to one of:

- **Off** — the plain order of the word and meaning matches.
- **Rerank** — the reranking model scores each match against the search and puts the best first.
- **Fuse** — the reranking model's scores added to the plain order.

On an index's page, **Try a search** says the order in plain words: "Best matches first: a second
model reread the top results" for Rerank (and "mixed with the usual order" for Fuse), or "In the
usual order" for Off.

All four use **Fuse** unless you choose otherwise; **Platform default** shows it. Rerank and Fuse
each cost a small amount for each search, paid from the wallet, and add a little time; through a
search key, a rerank is one published unit per search. Settings
shows the price and the most time they add. Past that time, a search answers in the plain order,
and a search whose time ran out before the reranking model ran is not charged for it.
Only an owner can change the settings.

The reranking model hides a match it scores too weak, but never one that holds a code you
typed: a search for `WT-30` or `flatMap()` always shows a document that contains it, exactly as
typed. Its score for a match is kept for a day, so the same search gives the same results, and
a repeat costs nothing for the matches it has scored. When the reranking model does not answer,
the search shows the plain order, including matches it would have hidden. When it scores every
match too weak, the search shows the plain order too, with the line "Showing the closest matches
by words and meaning.", rather than nothing; that search is charged, because the model ran.

When the platform moves an index to a new embedding model, the index keeps answering with the
old one until every document is embedded again.

## See how an index is searched

Open the index. **Searches, last 30 days** says how many searches it answered, how many found
nothing, and how long they took. It counts every door: your loops, your own searches, your
visitors and your search keys. A search across several indexes counts once in each.

The time is a range, not an exact figure: **under 250 ms**, or **2 s or more**. Under it, a list
gives each day that had a search; the days are UTC. When the counts cannot be read, the index
says so, and that is not a report that it had no searches.

## Delete an index

Open the index and press **Delete index**. Its documents and everything built from them are
removed.

## What an index costs

Crawling, browser renders and indexing are paid from the account's wallet, and each crawl
reserves its estimate before it starts. A crawl the wallet cannot cover does not start, and the
index says so.

Indexing is charged **per chunk**: a chunk is one section of a document, up to about 2,000
characters, and a typical document is 2 to 4 chunks. A long document costs more than a short one,
because it stores more. Indexing a document again with unchanged text costs nothing. The price
per 1,000 chunks is on the [pricing page](/pricing.md).

A document the wallet cannot cover waits as pending, and the index's page counts how many wait
for budget. You do not send them again: within about an hour after the wallet can pay, they index
on their own. While the wallet still holds less than one chunk's price, they keep waiting and
nothing is charged for them.

An owner of the account can give an index its own **monthly cap**, in dollars, on the index's
page, and remove it there. Members see the cap but cannot change it. The cap limits the index's
crawls, browser renders and indexing in one calendar month (UTC), beside the wallet: each one must
fit in both. A crawl whose estimate is more than the cap has left this month does not start, and
the index says so. A document the cap cannot cover waits as pending, and the index names the cap as
the reason. Within about an hour after the month turns or an owner raises the cap, those documents
index on their own. The cap counts only what the index spends while the cap is set, so a cap first
set in the middle of a month starts at zero. Searches are not limited by it.

A search by your loops, your visitors or you is charged a fixed amount for running it, whichever
way it searches, plus its query's embedding when it searches by meaning, plus the reranking when
it ran. A search through a search key is one published price that covers running it and its
query's embedding; a rerank, when the search asks for one and it sets the order, is added at its own
price. All of these prices per 1,000 are on the [pricing page](/pricing.md).

An index's **Spent this month** adds up its crawls, its indexing, and its share of every search
that searched it: your loops' runs, your own searches, your visitors and your search keys. A
search over several indexes is shared evenly among them. Each search is still paid once, from the
wallet.
