Inference

Latitude.sh AI Inference provides access to AI models through a unified, OpenAI-compatible API. Run open-weight text, vision, and reasoning models from DeepSeek, Z.ai, OpenAI, and Qwen.

AI Inference is in preview. Contact support to request access. Models, prices, and limits may change. See Availability during preview for what to expect.

Features

API keys: Generate and manage keys to authenticate your API requests
Endpoints: Browse available models with pricing, context length, and capabilities
Playground: Test models interactively before integrating them

Managing API keys

Access API keys

Create a key

Click Create key and provide a name for your key.

Copy your key

Copy the generated API key immediately. For security, the full key is only shown once.

API keys are prefixed with aitok_ and can be deleted at any time from the API keys page.

Browsing models

Access the endpoints page

Filter models

Click Create endpoint to search models by name and filter by provider or capability (text, vision, code, reasoning).

View model details

Select a model, then click Create endpoint to see its endpoint URL, model ID, provider, context length, and input/output pricing per million tokens.

Using the playground

Access the playground

Configure your request

Pick a model and an API key with the Ready status. If the Key value field appears, paste the full key. Click Try out, then use the settings button to adjust the system prompt, temperature, and max tokens.

Send a message

Type your prompt and send it to see the model’s response in real-time.

Making API requests

The AI Inference API is available at https://api.lsh.ai and is fully compatible with the OpenAI SDK. You can also make direct HTTP requests.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.lsh.ai/v1",
    api_key="YOUR_API_KEY",
)

response = client.chat.completions.create(
    model="deepseek-v4.1-flash",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

curl https://api.lsh.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "model": "deepseek-v4.1-flash",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.lsh.ai/v1",
  apiKey: "YOUR_API_KEY",
});

const response = await client.chat.completions.create({
  model: "deepseek-v4.1-flash",
  messages: [{ role: "user", content: "Hello!" }],
});
console.log(response.choices[0].message.content);

Replace YOUR_API_KEY with your API key. You can find available model IDs on the Endpoints page or with GET /v1/models. The SDK examples below use the client created above.

Send a User-Agent header with every request. The OpenAI SDKs and cURL send one. Python’s urllib sends a default that’s blocked with 403 and error code: 1010, so set your own when you use it.

Listing models

GET /v1/models lists the models currently offered. It requires a valid API key. Use a model’s id as the model value in your requests.

for model in client.models.list():
    print(model.id)

curl https://api.lsh.ai/v1/models \
  -H "Authorization: Bearer YOUR_API_KEY"

for await (const model of client.models.list()) {
  console.log(model.id);
}

Models

Model ID	Context length	Input	Cached input	Output	Capabilities
`deepseek-v4.1-flash`	1,048,576	$0.18	$0.005	$0.70	Text, vision, tools, reasoning
`glm-5.3`	131,072	$0.60	$0.12	$2.60	Text, tools, reasoning
`glm-5.3-flash`	131,072	$0.12	$0.025	$0.42	Text, tools, reasoning
`gpt-oss-120b`	32,768	$0.08	$0.04	$0.40	Text, tools, reasoning
`qwen3.8-27b`	131,072	$0.15	—	$1.90	Text, vision, tools, reasoning

Prices are in USD per million tokens. The context length counts input and output tokens together. GET /v1/models returns each model’s context_length and capabilities. Every response reports its token counts in usage. Input tokens include the model’s chat template, so they’re slightly higher than the tokens in your messages alone. When a request starts with the same tokens as a recent one, those input tokens come from the prompt cache. They’re reported in usage.prompt_tokens_details.cached_tokens and billed at the cached input price. qwen3.8-27b doesn’t use the prompt cache.

Reasoning

Every model reasons before it answers unless you turn reasoning off. The reasoning comes back apart from the answer, in message.reasoning, or in message.reasoning_content for glm-5.3 and glm-5.3-flash. When streaming, each chunk’s delta uses the same field. Reasoning tokens are billed as output tokens. Set reasoning_effort to choose how much a model reasons. To turn reasoning off, pass "chat_template_kwargs": {"enable_thinking": false}. A reasoning_effort value that a model doesn’t accept returns a 400 error.

Model	`reasoning_effort` values	Turn reasoning off
`deepseek-v4.1-flash`	`none`, `low`, `high`, `xhigh`, `max`	`none` or `enable_thinking: false`
`glm-5.3`	`none`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max`	Not supported
`glm-5.3-flash`	`none`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max`	Not supported
`gpt-oss-120b`	`low`, `medium`, `high`	Not supported
`qwen3.8-27b`	`none`, `low`, `medium`, `xhigh`	`none` or `enable_thinking: false`

glm-5.3 always reasons. It groups the values into three levels: none, minimal, low, and enable_thinking: false reason briefly, medium and high reason more, and xhigh, max, and the default reason the most.
glm-5.3-flash always reasons. It groups the values into three levels: none, minimal, low, and enable_thinking: false reason briefly, medium and high reason more, and xhigh, max, and the default reason the most.
gpt-oss-120b always reasons and ignores enable_thinking. Its usage.completion_tokens_details.reasoning_tokens is always 0. The reasoning tokens are still counted, and billed, in completion_tokens.

response = client.chat.completions.create(
    model="deepseek-v4.1-flash",
    messages=[{"role": "user", "content": "What is 17 * 23?"}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

curl https://api.lsh.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "model": "deepseek-v4.1-flash",
    "messages": [{"role": "user", "content": "What is 17 * 23?"}],
    "reasoning_effort": "low"
  }'

Tool calling

Pass your functions in tools. When the model decides to call one, the response has tool_calls instead of an answer. Run the function, append each result as a tool message, and send the conversation again to get the final answer. Every model on this page supports tool calling.

import json

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather in a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

messages = [{"role": "user", "content": "What's the weather in Paris?"}]
response = client.chat.completions.create(model="deepseek-v4.1-flash", messages=messages, tools=tools)
message = response.choices[0].message

if message.tool_calls:
    messages.append(message)
    for call in message.tool_calls:
        args = json.loads(call.function.arguments)
        result = {"city": args["city"], "temperature_c": 18}  # call your own function here
        messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)})
    response = client.chat.completions.create(model="deepseek-v4.1-flash", messages=messages, tools=tools)

print(response.choices[0].message.content)

curl https://api.lsh.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "model": "deepseek-v4.1-flash",
    "messages": [{"role": "user", "content": "What'"'"'s the weather in Paris?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather in a city",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }]
  }'

Structured outputs

To get JSON that matches a schema, set response_format to json_schema with your JSON Schema. The answer in message.content is a JSON string that follows the schema. Every model on this page supports structured outputs.

import json

response = client.chat.completions.create(
    model="deepseek-v4.1-flash",
    messages=[{"role": "user", "content": "What's the largest city in Japan?"}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "city",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"},
                    "country": {"type": "string"},
                    "population": {"type": "integer"},
                },
                "required": ["city", "country", "population"],
                "additionalProperties": False,
            },
        },
    },
)
print(json.loads(response.choices[0].message.content))

curl https://api.lsh.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "model": "deepseek-v4.1-flash",
    "messages": [{"role": "user", "content": "What'"'"'s the largest city in Japan?"}],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "city",
        "strict": true,
        "schema": {
          "type": "object",
          "properties": {
            "city": {"type": "string"},
            "country": {"type": "string"},
            "population": {"type": "integer"}
          },
          "required": ["city", "country", "population"],
          "additionalProperties": false
        }
      }
    }
  }'

Streaming responses

Set stream: true to receive the response as server-sent events while the model generates it. Each event is a data: line that carries a JSON chunk, and the stream ends with data: [DONE]. For chat completions, the last chunk before [DONE] has an empty choices array and a usage object with the token counts. It’s always sent, even if you don’t set stream_options.include_usage.

stream = client.chat.completions.create(
    model="deepseek-v4.1-flash",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)
for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="", flush=True)
    if chunk.usage:
        print(f"\nTotal tokens: {chunk.usage.total_tokens}")

curl https://api.lsh.ai/v1/chat/completions \
  -N \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "model": "deepseek-v4.1-flash",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true
  }'

const stream = await client.chat.completions.create({
  model: "deepseek-v4.1-flash",
  messages: [{ role: "user", content: "Hello!" }],
  stream: true,
});
for await (const chunk of stream) {
  if (chunk.choices.length) process.stdout.write(chunk.choices[0].delta.content ?? "");
  if (chunk.usage) console.log(`\nTotal tokens: ${chunk.usage.total_tokens}`);
}

Long generations

Use streaming for any response that can take longer than about 2 minutes to complete. A non-streaming response arrives only once it’s complete, so a non-streaming request that takes longer than about 2 minutes fails with a 504 error. A streaming response sends tokens as they’re generated, which keeps the connection active. Every request, including streams, is limited to 20 minutes. A stream that sends nothing for about 2 minutes, or reaches that limit, ends without data: [DONE].

Canceling requests

Close the connection to cancel a streaming request. Generation stops when the connection closes.

Python SDK: call stream.close(), or break out of a with block that wraps the stream.
Node.js SDK: break out of the for await loop, or call stream.controller.abort().
cURL: press Ctrl+C.

Request IDs

Every response, including errors, carries an x-request-id header. Include it when you contact support about a request. The OpenAI SDKs expose this header as response._request_id. When a request fails, read it from the error: error.request_id in Python, or error.requestID in Node.js. With cURL, add -i to print the response headers.

Rate limits

Each project has its own limits on each model:

Requests: 60 per minute.
Tokens: 250,000 per minute, input and output combined. A request’s tokens count when it finishes, so one long request can use up the budget and hold the next request until the budget refills.
Concurrent requests: a share of the model’s capacity, and at least 4. The error message states your current limit.

A request over a limit gets a 429 with the rate_limit_exceeded code and a Retry-After header. Wait that many seconds before you retry.

{
  "error": {
    "message": "rate limit exceeded",
    "type": "inference_error",
    "code": "rate_limit_exceeded",
    "detail": "This project is over its limit of 12 concurrent requests on this model."
  }
}

Contact support if your workload needs higher limits.

Errors

Errors use the OpenAI error format. Use the HTTP status to decide how to handle an error; error.code identifies the cause.

{
  "error": {
    "message": "model not found",
    "type": "inference_error",
    "code": "model_not_found"
  }
}

The table lists the common codes. Other codes can appear.

Status	Code	Meaning
`400`	`invalid_request`	The request is invalid, for example `model` is missing. Errors the model returns, such as a prompt longer than its context length, carry the model’s own message and code.
`401`	`invalid_api_key`	The API key is missing, invalid, or revoked. Check that the key has the Ready status.
`404`	`model_not_found`	The model isn’t offered. Check the ID with `GET /v1/models`.
`404`	`unsupported_endpoint`	The API doesn’t serve this path.
`405`	`method_not_allowed`	The path doesn’t accept this HTTP method. The `Allow` header lists the method it accepts.
`413`	`request_too_large`	The request body is larger than 15 MiB.
`429`	`rate_limit_exceeded`	Your project is over one of its rate limits. Retry after the `Retry-After` delay.
`429`	`capacity_exhausted`	The model is at capacity. Retry after the `Retry-After` delay.
`502`	`upstream_unreachable`, `upstream_error`	The connection to the model failed, and the request may have run.
`503`	`model_unavailable`, `cluster_unavailable`, `routing_unavailable`, `deployment_not_ready`, `model_not_deployed`, `identity_unavailable`, `admission_unavailable`, `upstream_unavailable`, `edge_overloaded`	The model can’t be served right now. Retry after the `Retry-After` delay, when present.
`504`	`504`, `timeout`, `upstream_timeout`	The model sent nothing for about 2 minutes, or the connection to it timed out. Use streaming for long generations.

Retrying requests

A response with x-should-retry: true is safe to retry, and x-should-retry: false means it isn’t. Every 5xx response carries the header: it’s false whenever the request may already have run, such as a 502 or a 504 timeout. The OpenAI SDKs follow this header and the Retry-After delay automatically. If you use another client, retry only responses marked x-should-retry: true, after the Retry-After delay. Never retry a stream that has already returned events. A stream that ends without data: [DONE] was interrupted, and a retry runs the whole request again.

Availability during preview

AI Inference is in preview, and some models run in a single location. While that location has a hardware failure, or while we update the model, requests to that model fail with a 503 until it’s back. Retry 503 responses after the Retry-After delay. The list of models can change. Check GET /v1/models for the models offered now.

Docs

Getting started

Compute

AI

Storage & data

Networking

Resource management

Platform settings

Usage & billing

Support

Legal

Features

Managing API keys

Browsing models

Using the playground

Making API requests

Listing models

Models

Reasoning

Tool calling

Structured outputs

Streaming responses

Long generations

Canceling requests

Request IDs

Rate limits

Errors

Retrying requests

Availability during preview

​Features

​Managing API keys

​Browsing models

​Using the playground

​Making API requests

​Listing models

​Models

​Reasoning

​Tool calling

​Structured outputs

​Streaming responses

​Long generations

​Canceling requests

​Request IDs

​Rate limits

​Errors

​Retrying requests

​Availability during preview

Features

Managing API keys

Browsing models

Using the playground

Making API requests

Listing models

Models

Reasoning

Tool calling

Structured outputs

Streaming responses

Long generations

Canceling requests

Request IDs

Rate limits

Errors

Retrying requests

Availability during preview