crawler warming up
A language model that reads what the crawlers read.
Crawlmind is a crypto-native corpus you can inspect before anyone trains on it. This page crawls live Solana token pages and on-chain memos, filters and deduplicates them, indexes them for search and trains a tokenizer on them, all in your browser.
Suggested:
01 What the crawlers read · live sample
The crawl, line by line.
Every 45 seconds the crawler fetches the newest Solana token profiles, boosts and community takeovers from DexScreener, the latest on-chain memos from the Solana Memo program, and trending coin names from Jupiter. Each document passes a quality filter, an exact SHA-256 dedup and a MinHash near-duplicate check. What survives becomes the corpus below.
- fetching sources
- fetching sources
- fetching sources
02 Tool · Index search
Search the corpus the model would see.
An inverted index over the kept documents, ranked with BM25 (k1 = 1.2, b = 0.75), the same scoring family classic search engines use. Results link to the real source page.
03 How a page becomes training data
Crawl, filter, dedupe, tokenize, train.
- Crawl. Fetch pages and on-chain text, follow their links.
- Filter. Drop machine IDs, spam and fragments too short to teach anything.
- Dedupe. Hash every document; MinHash catches the near-copies.
- Tokenize. Byte-pair merges turn text into the integers a model reads.
- Train. Loss falls as tokens flow; compute is roughly 6 × parameters × tokens.
04 Tool · Tokenizer lab
A tokenizer trained on this crawl, in your browser.
Byte-pair encoding learns the most frequent adjacent symbol pairs in the kept corpus and merges them, one rule at a time. Type anything to see how a crypto-native vocabulary splits it.
Waiting for the first crawl pass.
05 Tool · Dataset-mix designer
Design the training mix, see where it repeats.
Set a token budget and a weight for each source. The designer works out how many tokens each source contributes and how many passes (epochs) that means over its available pool. Past about four epochs, repeated data stops helping much. Pool sizes are labelled reference figures; the on-chain pool is derived from live Solana throughput.
| Source | Weight | Tokens | Pool | Epochs |
|---|
06 Tool · Tokens and cost to train
What would it take to train?
Training compute is about 6 × N × D floating-point operations for N parameters and D tokens. Divide by what your hardware sustains and you have GPU-hours; multiply by the hourly rate and you have the bill. The loss estimate uses the published Chinchilla fit and is a model, not a promise.
07 Tool · Crawl a wallet
Point the crawler at an address.
The crawler reads the address's 25 most recent mainnet transactions, four at a time, and maps every account and program they touch into a link graph, the way a web crawler maps pages and links. Connect your wallet to crawl your own.
The graph draws here.
08 Model card, honestly
What is live, what is a model, what does not exist yet.
The crawl stream, the search index, the tokenizer and the wallet crawl run on real mainnet and DexScreener data, fetched by your browser, right now. Solana throughput comes from getRecentPerformanceSamples.
Compute, cost and loss come from public formulas (6ND and the Chinchilla fit). Pool sizes in the mix designer are reference figures. Hardware prices are editable defaults, not quotes.
No Crawlmind model has been trained. There are no weights, no benchmarks and no partners. This site is the open notebook for the corpus, and it never holds funds or asks for keys.
09 Questions
Frequently asked.
Is the crawl really live?
Yes. Your browser calls DexScreener's public token-profile endpoints, Jupiter's pool data and Solana mainnet (through our read-only relay) on every pass. Nothing is cached on a server of ours.
Why are so many memos dropped?
Most on-chain memos are order IDs, hashes and bot references. They are text, but they teach a model nothing. The filter keeps documents with enough real words and drops machine identifiers.
How does near-duplicate detection work?
Each document is cut into three-word shingles and summarised by 64 MinHash values. Two documents whose signatures agree on 80 % or more are treated as near-copies and only the first is kept.
Is the tokenizer the one a real model would use?
It is a real byte-pair encoder trained on this page's corpus, so its merges reflect crypto text. Production tokenizers train on far more data and have tens of thousands of merges.
Where do the compute numbers come from?
Compute is 6 × parameters × tokens. Loss uses L = 1.69 + 406.4 / N^0.34 + 410.7 / D^0.28 from Hoffmann et al. (2022). Both are approximations and are labelled as a model.
What does connecting a wallet do?
It shares your public address so the wallet crawl can map your own transaction graph. Nothing is signed, sent or stored.