Skip to main content

Customers / End-User Budgets

Track spend, set budgets for your customers.

Tracking Customer Spend​

1. Make LLM API call w/ Customer ID​

Make a /chat/completions call, pass 'user' - First call Works

Make request with customer ID
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
        --header 'Content-Type: application/json' \
        --header 'Authorization: Bearer sk-1234' \ # 👈 YOUR PROXY KEY
        --data ' {
        "model": "azure-gpt-3.5",
        "user": "ishaan3", # 👈 CUSTOMER ID
        "messages": [
            {
            "role": "user",
            "content": "what time is it"
            }
        ]
        }'

The customer_id will be upserted into the DB with the new spend.

If the customer_id already exists, spend will be incremented.

2. Get Customer Spend​

Call /customer/info to get a customer's all up spend

Get customer spend
curl -X GET 'http://0.0.0.0:4000/customer/info?end_user_id=ishaan3' \ # 👈 CUSTOMER ID
        -H 'Authorization: Bearer sk-1234' \ # 👈 YOUR PROXY KEY

Expected Response:

Response
{
    "user_id": "ishaan3",
    "blocked": false,
    "alias": null,
    "spend": 0.001413,
    "allowed_model_region": null,
    "default_model": null,
    "litellm_budget_table": null
}

Setting Customer Budgets​

Set customer budgets (e.g. monthly budgets, tpm/rpm limits) on haimaker Proxy

Default Budget for All Customers​

Apply budget limits to all customers without explicit budgets. This is useful for rate limiting and spending controls across all end users.

Step 1: Create a default budget

Create default budget
curl -X POST 'http://localhost:4000/budget/new' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{
    "max_budget": 10,
    "rpm_limit": 2,
    "tpm_limit": 1000
}'

Step 2: Configure the default budget ID

config.yaml
litellm_settings:
  max_end_user_budget_id: "budget_id_from_step_1"

Step 3: Test it

Make request with customer ID
curl -X POST 'http://localhost:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Hello"}],
    "user": "my-customer-id"
}'

The customer will be subject to the default budget limits (RPM, TPM, and $ budget). Customers with explicit budgets are unaffected.

Quick Start​

Create / Update a customer with budget

Create New Customer w/ budget

Create customer with budget
curl -X POST 'http://0.0.0.0:4000/customer/new'         
    -H 'Authorization: Bearer sk-1234'         
    -H 'Content-Type: application/json'         
    -d '{
        "user_id" : "my-customer-id",
        "max_budget": "0", # 👈 CAN BE FLOAT
    }'

Test it!

Test customer budget
curl -X POST 'http://localhost:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-D '{
    "model": "mistral",
    "messages": [
        {
        "role": "user",
        "content": "What'\''s the weather like in Boston today?"
        }
    ],
    "user": "ishaan-jaff-48"
}

Assign Pricing Tiers​

Create and assign customers to pricing tiers.

1. Create a budget​

  • Go to the 'Budgets' tab on the UI.
  • Click on '+ Create Budget'.
  • Create your pricing tier (e.g. 'my-free-tier' with budget $4). This means each user on this pricing tier will have a max budget of $4.

2. Assign Budget to Customer​

In your application code, assign budget when creating a new customer.

Just use the budget_id used when creating the budget. In our example, this is my-free-tier.

Assign budget to customer
curl -X POST 'http://localhost:4000/customer/new' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-D '{
    "user_id": "my-customer-id",
    "budget_id": "my-free-tier" # 👈 KEY CHANGE
}

3. Test it!​

Test with curl
curl -X POST 'http://localhost:4000/customer/new' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-D '{
    "user_id": "my-customer-id",
    "budget_id": "my-free-tier" # 👈 KEY CHANGE
}