Overview of OpenAI GPT models with a focus on image input capabilities
GPT()
The GPT class is designed for text-to-text interactions, utilizing models from the Generative Pre-trained Transformer series. These models are adept at understanding and generating text based on the input provided.
Takes a text query and returns the modelโs text response.
GPTVision()
The GPTVision class extends the capabilities of GPT to include handling image inputs, making it suitable for image-to-text tasks.
This class is part of the multimodal models that can process both text and images.
GPT-4o and GPT-4v are the leading Vision Language models and are capable of understanding relatively fine details in images.
Although when scaling to datasets of millions of images, these models can become prohibitively expensive.
Generally, GPT-4o is the best model to use for image-to-text tasks, and is cheaper than GPT-4v.
The image is a black-and-white photograph featuring a man in a suit seemingly interacting with the Eiffel Tower. The man stands in the foreground with one arm extended, appearing to be touching the top of the Eiffel Tower with his hand. This perspective trick creates the illusion that he is much larger than the tower itself. The background shows the full view of the Eiffel Tower, with the base and the surrounding area visible. The photo is taken in a way that makes it look like the man is playfully leaning on or manipulating the tower due to the forced perspective technique. The man is dressed in a suit and tie, looking down at his hand with a thoughtful or playful expression. The setting is likely the Champ de Mars, a large public greenspace near the Eiffel Tower in Paris, France.
wget...
from overeasy import*from PIL import Image
workflow = Workflow([
BinaryChoiceAgent("Do the Celtics have possession of the ball?", model=GPTVision(model="gpt-4o"))])
image=Image.open("bball.png")
result, graph = workflow.execute(image)print(result[0].data.class_names)
Q: Do the Celtics have possession of the ball? A: No
Many GPT Visual models have relatively strict guard rails around inputs/photos of people so your requests may get screened out if they are related to these topics.
These models also generally struggle with spatial reasoning(i.e. is object A in front of object B?).