Skip to main content

​
CLIP Class Documentation

​
Overview

The CLIP class is designed to perform image classification using the CLIP (Contrastive Language-Image Pretraining) model. It leverages a pre-trained model from OpenAI to classify images into predefined categories based on textual descriptions.

​
Methods

​
__init__(self, model_card: str = "openai/clip-vit-large-patch14")

Constructor for the CLIP class.
  • Parameters:
    • model_card: A string identifier for the pre-trained CLIP model to be used. Default is "openai/clip-vit-large-patch14".

​
load_resources(self)

Loads the necessary resources for the model including the processor and the model itself from the specified model_card.

​
release_resources(self)

Releases the loaded resources to free up memory. This is particularly useful when working with limited memory resources or in a GPU environment to clear the cache.

​
classify(self, image: Image.Image, classes: list) -> Detections

Classifies an image into one of the provided classes.
  • Parameters:
    • image: An instance of Image.Image to be classified.
    • classes: A list of strings representing the class labels to classify the image against.
  • Returns:
    • A Detections object containing:
      • xyxy: Bounding box coordinates (always zero as CLIP does not provide localization).
      • class_ids: Array containing the index of the predicted class.
      • confidence: Array containing the confidence score of the prediction.
      • classes: Array of class labels provided.
      • detection_type: Type of detection, which is DetectionType.CLASSIFICATION in this case.

​
Example Usage

wget -O basketball.png "https://github.com/overeasy-sh/overeasy/blob/main/examples/basketball.png?raw=true"

from overeasy import *
from PIL import Image
workflow = Workflow([
  ClassificationAgent(classes=["basketball", "soccer", "football"], model=CLIP()),
])

image = Image.open("./basketball.png")
results, graph = workflow.execute(image)
workflow.visualize(graph)
For more information on OwlV2, refer to the original paper.
⌘I