File Vectors

File Vectors

File Vectors turns a pinned text file into vector embeddings, so you can search your files by meaning (semantic search) instead of by exact keywords. You can create the vectors while you pin a file, or later for a file you already pinned. A query returns your pin objects directly, so you don't need to keep your own table that maps vectors back to files. The original file stays on IPFS and is not changed. AIOZ Pin makes the embeddings with the BAAI/bge-m3 model, which supports more than 100 languages.

Pin a text file and create its vectors in one request:

curl --location --request POST 'https://api.aiozpin.network/api/pinning/' \
--header 'pinning_api_key: KEY' \
--header 'pinning_secret_key: SECRET' \
--form 'file=@"./notes.md"' \
--form 'metadata="{\"name\": \"notes.md\"}"' \
--form 'options="{\"vectorize\": true}"'

When the file's vectorize_status is completed, query your file vectors with plain text:

curl --location --request POST 'https://api.aiozpin.network/api/v2/vectorize/query' \
--header 'pinning_api_key: KEY' \
--header 'pinning_secret_key: SECRET' \
--header 'Content-Type: application/json' \
--data-raw '{"text": "how do I store files on IPFS"}'

Limits

Only UTF-8 text files can be vectorized. The file type must be one of:

  • text/* (for example text/plain, text/markdown, text/csv, text/html)
  • application/json, application/xml, application/yaml, application/x-yaml
  • application/javascript, application/typescript, application/csv
  • any type ending with +json or +xml

Rich documents (PDF, DOCX, images, audio, video), CAR files, and folders are not supported. A request for an unsupported file returns 400 Unsupported file type for vectorize: <mime type>.

When you upload a file, AIOZ Pin detects its type from the content, not from the file name. Most text files, including Markdown, CSV, and JSON, are stored as text/plain; charset=utf-8. When you pin by CID, the type comes from the IPFS gateway.

The text is embedded as it is. HTML and XML tags are not removed, and bytes that are not valid UTF-8 are dropped.

Other limits:

  • Each account has one vector index, and a query only searches files in your account.
  • Long files are split into parts of about 2,000 characters. A file ranks by its best-matching part and appears once in the results.
  • A query returns up to 100 files, 10 by default.
  • Results have no similarity score and no matched text.

Permissions

All File Vectors endpoints require your pinning API key and secret key, and the key needs the admin scope or the pin_file_to_ipfs scope. Otherwise the API returns 401: API Key don't have permissions to perform this action.

When you vectorize on upload or on pin by CID, the pin endpoint's own scope applies: pin_file_to_ipfs for a file upload and pin_by_hash for pin by CID.

Vector Status

Each file has one vector job. Pin objects show its state in vectorize_status. Read it with Get Pin Details by pinID or List Pins of User.

StatusMeaning
not_requestedNo vectors were requested for this file.
queuedWaiting to run. A file pinned with vectorize: true stays here until pinning finishes.
processingThe file is being read, embedded, and stored.
completedThe file appears in query results.
failedThe last attempt failed. The retry rules are below this table.
deletedThe file's vectors were deleted with Delete File Vectors. After they are cleaned up, usually within a day, the status goes back to not_requested.

When an attempt fails, the API retries the file in the background, up to 5 attempts in total. The wait between attempts doubles each time, starting at 30 seconds. If the file is still failed after that, call Vectorize a File to try again. A file with no text, such as an empty file, always fails.

Vectorizing Files

There are three ways to vectorize a file:

  1. Vectorize on Upload with the vectorize pin option.
  2. Vectorize on Pin by CID with the pin_options.vectorize option.
  3. Vectorize a File that is already in your account.

Then you can query your file vectors and delete the vectors of a file.

With the first two, the response contains vectorize_status: "queued", and the file is vectorized in the background once it is pinned. Check progress with Get Pin Details by pinID until vectorize_status is completed.

Vectorize on Upload

Send the options form field to POST /api/pinning/:

{
  "cid_version": 1,
  "wrap_with_directory": false,
  "vectorize": true
}
curl --location --request POST 'https://api.aiozpin.network/api/pinning/' \
--header 'pinning_api_key: KEY' \
--header 'pinning_secret_key: SECRET' \
--form 'file=@"./notes.md"' \
--form 'metadata="{\"name\": \"notes.md\"}"' \
--form 'options="{\"vectorize\": true}"'

Response:

{
    "data": {
        "id": "string",
        "root_hash": "string",
        "cid": "string",
        "size": number,
        "user_id": "string",
        "pinned": true,
        "is_pin_by_hash": false,
        "is_dir": false,
        "metadata": {
            "name": "notes.md",
            "type": "text/plain; charset=utf-8"
        },
        "status": "string",
        "vectorize_status": "queued"
    },
    "status": "success"
}

Rules when vectorize is true:

  • Upload one file only. Multiple files or wrap_with_directory: true returns 400 Vectorize only supports a single text file upload.
  • The file type is checked before pinning. A non-text file returns 400 Unsupported file type for vectorize.
  • CAR files return 400 CAR files are not supported for vectorize.
  • Chunked upload (/api/v2/pinning/completeUpload?vectorize=true) returns 400 Vectorize is not supported for chunked upload completion. Pin the file first, then call Vectorize a File.
  • If this file is already pinned in your account, the API returns 409 Pin with that hash already exists. Use Vectorize a File instead.

Vectorize on Pin by CID

Add pin_options.vectorize to the pinByHash body:

{
    "hash_to_pin": "CID",
    "metadata": {
        "name": "string"
    },
    "pin_options": {
        "vectorize": true
    }
}
curl --location --request POST 'https://api.aiozpin.network/api/pinning/pinByHash/' \
--header 'pinning_api_key: KEY' \
--header 'pinning_secret_key: SECRET' \
--header 'Content-Type: application/json' \
--data-raw '{
    "hash_to_pin": "Qmc1135ziMvmFG534i75E8HpJoLqzLzgKBxfjBV9cBsMAs",
    "metadata": { "name": "name-ipfs-hash" },
    "pin_options": { "vectorize": true }
}'

Before pinning, the API reads the file type from the IPFS gateway. This check can be slow if the CID is hard to find on the network.

  • An invalid CID returns 400 Invalid cid.
  • An unsupported type returns 400 Unsupported file type for vectorize.
  • If the gateway cannot return the file, the request fails with 500 and the CID is not pinned.

Pinning by CID can take time, so vectorize_status stays queued until the file is pinned. Vectorizing starts after that.

Vectorize a File

Generate embeddings for a file that is already pinned. The file is identified by its CID and must belong to your account.

POSThttps://api.aiozpin.network/api/v2/vectorize/files/:cid

Parameters

cid* CID of the pinned file

Headers

pinning_api_key* PINNING-API-KEY
pinning_secret_key* PINNING-SECRET-KEY

Response

200: OK
{
    "data": {
        "id": "string",
        "pin_id": "string",
        "user_id": "string",
        "cid": "string",
        "index_id": "string",
        "status": "completed",
        "model": "BAAI/bge-m3",
        "model_version": "bge-m3",
        "dimensions": 1024,
        "chunk_count": 3,
        "content_hash": "string",
        "attempt_count": 1,
        "created_at": "2023-01-01T11:11:11.111111Z",
        "started_at": "2023-01-01T11:11:11.111111Z",
        "completed_at": "2023-01-01T11:11:12.111111Z"
    },
    "status": "success"
}

The request has no body. The call waits until the whole file is read, embedded, and stored, so a large file can take minutes. If the request times out, check vectorize_status with Get Pin Details by pinID. A failed attempt is retried in the background.

Behavior to know:

  • The file must be fully pinned (pinned: true and status is pinned). Otherwise the API returns 400 File is not pinned yet.
  • A CID that is not in your account returns 404 Pin not found.
  • It is safe to call again. If the file is already completed with the same CID, the API returns the existing job without doing the work again.
  • If vectorizing fails, the response is 500 with the message Something went wrong, and vectorize_status becomes failed. See Vector Status for retries.
  • Calling it on a file whose vectors you deleted with Delete File Vectors adds the file back to query results.

Response fields:

FieldDescription
idVector job ID.
pin_idID of the pin that was vectorized.
user_idYour user ID.
cidCID of the file.
index_idYour vector index. Each account has its own index.
statusAlways completed in a successful response.
model, model_versionEmbedding model, BAAI/bge-m3.
dimensionsLength of each vector (1024).
chunk_countNumber of text parts stored for the file. Long files are split into parts of about 2,000 characters.
content_hashSHA-256 hash of the file content.
attempt_countNumber of times the file has been processed.
created_at, started_at, completed_atJob timestamps.
curl --location --request POST 'https://api.aiozpin.network/api/v2/vectorize/files/Qmc1135ziMvmFG534i75E8HpJoLqzLzgKBxfjBV9cBsMAs' \
--header 'pinning_api_key: KEY' \
--header 'pinning_secret_key: SECRET'

Query File Vectors

Search your vectorized files with a natural language query. Results are ranked by how close their meaning is to the query.

POSThttps://api.aiozpin.network/api/v2/vectorize/query

Headers

pinning_api_key* PINNING-API-KEY
pinning_secret_key* PINNING-SECRET-KEY
Content-Type* application/json

Body

text* Search text
limit Maximum number of files to return (default 10, maximum 100). A value above 100 returns a 400 error (limit must not exceed 100)
filter Optional object. Only pin_id is supported, for example {"pin_id": "<pin id>"} to search one file.

Response

200: OK
{
    "data": {
        "query": "how do I store files on IPFS",
        "count": 1,
        "results": [
            {
                "id": "string",
                "root_hash": "string",
                "cid": "string",
                "size": number,
                "user_id": "string",
                "date_pinned": "2023-01-01T11:11:11.111111Z",
                "pinned": true,
                "is_pin_by_hash": false,
                "is_dir": false,
                "metadata": {
                    "name": "notes.md",
                    "type": "text/plain; charset=utf-8"
                },
                "status": "pinned",
                "vectorize_status": "completed"
            }
        ]
    },
    "status": "success"
}

Behavior to know:

  • Results are pin objects, ordered from the most similar to the least similar. Long files are split into parts of about 2,000 characters, and a file ranks by its best-matching part. A file appears once in the results. Use the cid to fetch or download the file.
  • The response has no similarity score and no matched text. Search returns the closest files even when none of them is about the query.
  • count can be lower than limit even when you have more vectorized files, because several of the best-matching parts can come from the same file.
  • Only files with vectorize_status: completed are returned. A file whose vectors you deleted with Delete File Vectors disappears from the results right away.
  • Unpinning a file does not delete its vectors, so results can include files with pinned: false. Call Delete File Vectors to remove them.
  • filter matches the metadata stored with each vector, which is only pin_id. {"pin_id": "<pin id>"} limits the search to that one file. Filters on the file name or keyvalues return no results.
  • text is required and cannot be blank. limit must be 100 or less. A value above 100 returns 400 with limit must not exceed 100. If limit is omitted or 0 or below, the default 10 is used.
curl --location --request POST 'https://api.aiozpin.network/api/v2/vectorize/query' \
--header 'pinning_api_key: KEY' \
--header 'pinning_secret_key: SECRET' \
--header 'Content-Type: application/json' \
--data-raw '{
    "text": "how do I store files on IPFS",
    "limit": 5
}'

Delete File Vectors

Delete the vectors of a file, which removes it from query results. This does not unpin or delete the file. To remove the file itself, see Remove Files (Unpin).

DELETEhttps://api.aiozpin.network/api/v2/vectorize/files/:cid

Parameters

cid* CID of the pinned file

Headers

pinning_api_key* PINNING-API-KEY
pinning_secret_key* PINNING-SECRET-KEY

Response

200: OK
{
    "data": {
        "message": "Vectorize job marked for deletion",
        "cid": "string"
    },
    "status": "success"
}

The file no longer appears in Query File Vectors results right after this call. Its vectorize_status becomes deleted, and the vectors are deleted in the background, usually within a day. To add the file back, call Vectorize a File again.

The call returns 200 even if the file was never vectorized. A CID that is not in your account returns 404.

curl --location --request DELETE 'https://api.aiozpin.network/api/v2/vectorize/files/Qmc1135ziMvmFG534i75E8HpJoLqzLzgKBxfjBV9cBsMAs' \
--header 'pinning_api_key: KEY' \
--header 'pinning_secret_key: SECRET'

See Also

  • Pinning - pin a file with the vectorize option, and read vectorize_status.
  • ApiKeys - the pin_file_to_ipfs scope required by these endpoints.
  • Limit - rate limits.