Skip to main content

Using Vision Models

Quick Start​

Example passing images to a model.

Python​

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api.haimaker.ai/v1"
)

response = client.chat.completions.create(
    model="openai/gpt-4-vision-preview",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What's in this image?"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-702702702nature-702702702702center.jpg/1280px-Gfp-wisconsin-madison-the-nature-center.jpg"
                    }
                }
            ]
        }
    ],
)

print(response.choices[0].message.content)

cURL​

curl https://api.haimaker.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "model": "openai/gpt-4-vision-preview",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What'\''s in this image?"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-center.jpg/1280px-Gfp-wisconsin-madison-the-nature-center.jpg"
            }
          }
        ]
      }
    ]
  }'

Using Different Vision Models​

Claude​

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api.haimaker.ai/v1"
)

response = client.chat.completions.create(
    model="anthropic/claude-3-7-sonnet-latest",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What's in this image?"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-center.jpg/1280px-Gfp-wisconsin-madison-the-nature-center.jpg"
                    }
                }
            ]
        }
    ],
)

print(response.choices[0].message.content)

Gemini​

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api.haimaker.ai/v1"
)

response = client.chat.completions.create(
    model="gemini/gemini-1.5-pro",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What's in this image?"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-center.jpg/1280px-Gfp-wisconsin-madison-the-nature-center.jpg"
                    }
                }
            ]
        }
    ],
)

print(response.choices[0].message.content)

Explicitly Specify Image Type​

If you have images without a mime-type, or if the API is incorrectly inferring the mime type of your image, you can set this explicitly via the format param.

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api.haimaker.ai/v1"
)

response = client.chat.completions.create(
    model="anthropic/claude-3-7-sonnet-latest",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What's in this image?"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://example.com/my-image",
                        "format": "image/jpeg"
                    }
                }
            ]
        }
    ],
)

print(response.choices[0].message.content)

The format parameter is used for APIs that support specifying mime-type (e.g., Anthropic, Bedrock, Vertex AI). For others (e.g., OpenAI), it will be ignored.

Spec​

"image_url": str

OR

"image_url": {
  "url": "url OR base64 encoded str",
  "detail": "openai-only param",
  "format": "specify mime-type of image"
}