Search Indexes

Reactor search makes your content searchable: a website, files you upload, or the records of one of your collections. It is a product of its own. You can use it with no loop, and a loop can use it too.

Every search reads one or more search indexes. You create an index once, on the account, on the Indexes page. Then four kinds of searcher can search it:

• You, on the console’s Indexes page. See Search your indexes yourself.
• Your loops. A loop names the indexes its runs may search. See Letting a Loop Search.
• Your visitors, through a published loop’s search box or its search API. See Let your visitors search.
• Your servers, with a search key and no loop. See Search from your server.

How to write a search is on Writing a Search: words, an exact phrase, words with a typo, a pattern, and words or meaning.

Create an index

On Indexes, press New index and choose what it is built from. That choice never changes.

Every index has a name for people, such as “Product documentation”, and an id that a loop uses to name it, such as docs. The id is a lower-case letter, then lower-case letters, digits or underscores. The form fills in the id from the name, for example shop_hours from “Shop hours”, until you type your own.

A new index is untrusted: what it finds counts as outside text, which limits what a loop that searches it may also do (the Rule of Two). An owner of the account can mark it trusted on its page, with Mark trusted…, once it exists; a member sees the mark and cannot change it. Mark an index trusted only for content you control. Trust also changes what a loop that searches the index may do; see What an index lets a loop do.

One choice applies to every kind:

• Search by meaning as well as by words. On by default. With it on, the index finds a passage that says the same thing in other words, as well as the exact words.

A site crawl

Give the crawl an address, such as https://docs.example.com/guide/. The crawl reads that host under that path, and nothing outside it. It reads the site’s robots.txt and sitemap and follows the links it finds inside that scope. Choose how often it fetches the site again, daily or weekly, and at most how many pages it stores: 2,000 by default, and at most 10,000.

The first crawl usually starts within an hour of creating the index; Fetch now starts it at once. Until then the list reads First crawl soon, never Ready. An index of files or of a collection’s records reads Empty in the list while it holds nothing, and shows on its own page how many documents it holds and how many are still being indexed. A page that only shows its text after its scripts run, as in a single-page app, is opened in a browser so that its text is found. The crawl reaches public sites only, and sends no login.

A crawled page takes its title from the first heading of its text, so a search result shows the page’s own name and not the site’s name repeated. A page with no heading keeps the title the site gives it.

A crawled page keeps its content and leaves out the site around it: the navigation, the header and footer, the sidebar, and furniture that documentation sites put beside the text, such as an “On this page” list, a version badge, a “Was this page helpful?” box or a “Last modified” line. A footer the site writes as ordinary text in the page, with nothing marking it as one, stays in.

The index shows its last crawl in words: whether it is crawling now or finished, how many pages it fetched, stored, rendered in a browser and failed, and what it cost. A crawl that runs out of money stops, keeps the pages it fetched, and says so.

Each account has a monthly crawl allowance, set by the platform: about 180,000 pages a month, across all of the account’s crawl indexes. When it is used up, a crawl that is running stops and keeps the pages it fetched, a new crawl does not start, and the index says monthly crawl allowance. It renews on the 1st (UTC). Fetch now refuses in the same words, and the press does not count. If your account needs more, contact us.

Fetch now crawls the site again at once, when you changed pages and do not want to wait for the schedule. It starts the same crawl the schedule runs, and costs the same. An account can press it 6 times a day (the day ends at 00:00 UTC); a press that starts nothing does not count. It starts nothing while a crawl of that index runs, and says when that crawl started. The schedule keeps its own crawls either way.

A crawl that finishes removes the pages it no longer reaches: a page now outside the address, a page the site skips or refuses to crawlers, and a second address that serves the same text as a page it kept, so a search never finds one page twice. A page that fails to load once keeps its place. A crawl that stops early, or that reaches its page limit, removes nothing.

A crawl that reaches the end of the site but keeps no page also removes nothing. This happens when the site shows a placeholder or is down. The index says crawled: no page kept, keeps what earlier crawls stored, and Home names it until a crawl keeps a page again.

Files you upload

Open the index and press Upload a file. The index takes Markdown, plain text and HTML files, and PDFs with a text layer. A scanned PDF has no text layer, so the index refuses it and says why. Each document shows its state: pending until it can be searched, then indexed. To take a file out of the index, press Remove… beside it, then Remove it. The console asks first because a removed file cannot be put back: upload it again to restore it.

A file added through the API can also name its source: the address it came from, such as the web page it copies. A search result then shows that address, so an answer can cite where the text came from. The address must start with https:// or http://.

A collection’s records

Name the collection and the fields to index, such as company, notes. The index starts with the records already stored, then follows every record a run writes or deletes.

To show records that describe the same thing as one result, name the field that identifies it under Same item when this field matches, such as sku: two listings of one product then show as one search result. It is a text or number field the collection declares, and not one that holds what people typed. A record without a value in it is still its own result. You choose this field when you create the index and cannot change it later; to fold on another field, create a new index.

Records can hold personal data. With search by meaning on, their text is sent to the embedding provider.

Search from your server

An upload or crawl index can also be searched with no loop, by a server of yours. On the index’s page, an owner mints a search key under For a server of yours: search keys; the key searches that index and does nothing else. Each search is charged at the published price per search. The request, the answer and the refusals are on Search API. A collection index has no search keys: its records can hold personal data.

Search your indexes yourself

On Indexes, press Search every index above the list to search all your indexes at once, without opening one. An index’s own Try a search tab opens the same search, starting on that index. Type what you are looking for, then choose:

• Find — the words, the exact phrase, the words allowing typos, or a pattern (one term). Writing a Search shows each with an example.
• In — every index, or one.
• By — Both, Words only or Meaning only. Words find an exact name, such as a command-line flag. Meaning finds a passage that says the same thing in other words. Both ranks the two together, as a loop’s own searches do. Every search is charged a small fixed amount for running it. A search by meaning also embeds your query once, at the index’s embedding price. Your wallet pays both.

Each result names its index and its title, and says when the index is untrusted. Its passage reads as text, and code in it keeps its lines, in a monospace box of its own that ends with ”…” when the passage stops inside the code. Press a result’s title to read the document from that place.

A document is one result, at its best-matching section. When other sections of the same page also matched, the result says so, for example +2 more sections; its title opens the whole page. A page crawled into two of your indexes is one result too, and so are records with the same value in their index’s Same item when this field matches field. A run’s agent gets the same results, and is told that the rest of each page can be read.

How results are ordered

Search order on Settings says in one line what your searches use now, such as “Every search uses Fuse”. Under More it chooses how searches are ordered, separately for four kinds of search: A run’s agent, Your search on Indexes, Your visitors on your site, and Your search keys (a server of yours, through the Search API). Each one is set to one of:

• Off — the plain order of the word and meaning matches.
• Rerank — the reranking model scores each match against the search and puts the best first.
• Fuse — the reranking model’s scores added to the plain order.

On an index’s page, Try a search says the order in plain words: “Best matches first: a second model reread the top results” for Rerank (and “mixed with the usual order” for Fuse), or “In the usual order” for Off.

All four use Fuse unless you choose otherwise; Platform default shows it. Rerank and Fuse each cost a small amount for each search, paid from the wallet, and add a little time; through a search key, a rerank is one published unit per search. Settings shows the price and the most time they add. Past that time, a search answers in the plain order, and a search whose time ran out before the reranking model ran is not charged for it. Only an owner can change the settings.

The reranking model hides a match it scores too weak, but never one that holds a code you typed: a search for WT-30 or flatMap() always shows a document that contains it, exactly as typed. Its score for a match is kept for a day, so the same search gives the same results, and a repeat costs nothing for the matches it has scored. When the reranking model does not answer, the search shows the plain order, including matches it would have hidden. When it scores every match too weak, the search shows the plain order too, with the line “Showing the closest matches by words and meaning.”, rather than nothing; that search is charged, because the model ran.

When the platform moves an index to a new embedding model, the index keeps answering with the old one until every document is embedded again.

See how an index is searched

Open the index. Searches, last 30 days says how many searches it answered, how many found nothing, and how long they took. It counts every door: your loops, your own searches, your visitors and your search keys. A search across several indexes counts once in each.

The time is a range, not an exact figure: under 250 ms, or 2 s or more. Under it, a list gives each day that had a search; the days are UTC. When the counts cannot be read, the index says so, and that is not a report that it had no searches.

Delete an index

Open the index and press Delete index. Its documents and everything built from them are removed.

What an index costs

Crawling, browser renders and indexing are paid from the account’s wallet, and each crawl reserves its estimate before it starts. A crawl the wallet cannot cover does not start, and the index says so.

Indexing is charged per chunk: a chunk is one section of a document, up to about 2,000 characters, and a typical document is 2 to 4 chunks. A long document costs more than a short one, because it stores more. Indexing a document again with unchanged text costs nothing. The price per 1,000 chunks is on the pricing page.

A document the wallet cannot cover waits as pending, and the index’s page counts how many wait for budget. You do not send them again: within about an hour after the wallet can pay, they index on their own. While the wallet still holds less than one chunk’s price, they keep waiting and nothing is charged for them.

An owner of the account can give an index its own monthly cap, in dollars, on the index’s page, and remove it there. Members see the cap but cannot change it. The cap limits the index’s crawls, browser renders and indexing in one calendar month (UTC), beside the wallet: each one must fit in both. A crawl whose estimate is more than the cap has left this month does not start, and the index says so. A document the cap cannot cover waits as pending, and the index names the cap as the reason. Within about an hour after the month turns or an owner raises the cap, those documents index on their own. The cap counts only what the index spends while the cap is set, so a cap first set in the middle of a month starts at zero. Searches are not limited by it.

A search by your loops, your visitors or you is charged a fixed amount for running it, whichever way it searches, plus its query’s embedding when it searches by meaning, plus the reranking when it ran. A search through a search key is one published price that covers running it and its query’s embedding; a rerank, when the search asks for one and it sets the order, is added at its own price. All of these prices per 1,000 are on the pricing page.

An index’s Spent this month adds up its crawls, its indexing, and its share of every search that searched it: your loops’ runs, your own searches, your visitors and your search keys. A search over several indexes is shared evenly among them. Each search is still paid once, from the wallet.