내 컴퓨터에서 llm으로 이미지를 분석해 보쟈! (feat. qwen2-VL)

qwen2의 모델들 맛보기를 계속하고있습니다!!^^

2024.10.03 - [데이터&AI/LLM] - 네가 그렇게 수학을 잘하니? Qwen2.5-Math (feat. 오픈소스 LLM)

네가 그렇게 수학을 잘하니? Qwen2.5-Math (feat. 오픈소스 LLM)

오늘은 이미지 분석 모델인 qwen2-VL에 대하여 알아보겠습니다!!

https://github.com/QwenLM/Qwen2-VL?tab=readme-ov-file

GitHub - QwenLM/Qwen2-VL: Qwen2-VL is the multimodal large language model series developed by Qwen team, Alibaba Cloud.

 

 

1. 모델 설치

 > vl은 이미지 가공이기에 관련 패키지 install이 필요했습니다!!

아래와 같이 설치해줍니다!!

 

pip install qwen-vl-utils[decord]

 

 

그리고 나서 python 에서 임포트 및 모델 설치를 진행합니다!! 

qwen2-VL의 2B / instruct 모델을 사용해봅니다!!

from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from qwen_vl_utils import process_vision_info

# default: Load the model on the available device(s)
model = Qwen2VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2-VL-2B-Instruct", torch_dtype="auto", device_map="auto"
)

 

 

2. 이미지 분석해보기!

 저는 gpt로 그린 아래 그림의 분석을 요청해보겠습니다!!!

 

 

"/home/bongo/porter_notebook/llm/qwen/siba.jpg"에 이미지를 저장했고!!

아래 코드를 실행하면요!?

# default processer
processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-2B-Instruct")

# The default range for the number of visual tokens per image in the model is 4-16384.
# You can set min_pixels and max_pixels according to your needs, such as a token range of 256-1280, to balance performance and cost.
# min_pixels = 256*28*28
# max_pixels = 1280*28*28
# processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct", min_pixels=min_pixels, max_pixels=max_pixels)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "/home/bongo/porter_notebook/llm/qwen/siba.jpg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

# Preparation for inference
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")

# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

 

우와~~ 설명을 잘해주네요!!

["The image depicts a Shiba Inu dog running through a snowy landscape. The dog appears to be in motion, with its front paws lifted off the ground and its body leaning forward. The background is blurred, but it shows a snowy environment with snowflakes falling, indicating a cold, wintry setting. The dog's fur is a mix of light brown and white, and it has a cheerful expression on its face, with its mouth open and eyes wide. The overall scene conveys a sense of joy and playfulness in the dog's demeanor."]

 

과연 한국어는!??

# default processer
processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-2B-Instruct")

# The default range for the number of visual tokens per image in the model is 4-16384.
# You can set min_pixels and max_pixels according to your needs, such as a token range of 256-1280, to balance performance and cost.
# min_pixels = 256*28*28
# max_pixels = 1280*28*28
# processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct", min_pixels=min_pixels, max_pixels=max_pixels)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "/home/bongo/porter_notebook/llm/qwen/siba.jpg",
            },
            {"type": "text", "text": "한국어로 이미지 분석해줘"},
        ],
    }
]

# Preparation for inference
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")

# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

 

어머나!! 한국어까지 이렇게 잘하다니,,,, 대단합니다!!

 

 

마지막!!,, 여러가지 언어도로 잘할수 있을까요??

정말 깜짝 놀랐습니다!! 멋지군요!!!

 

3. OCR 기능도 잘할까????

 이제, 한글 OCR 기능도 테스트해보겠습니다!!

삼성전자의 2024 2Q 분기보고서 이미지를 해볼게요!!

# default processer
processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-2B-Instruct")

# The default range for the number of visual tokens per image in the model is 4-16384.
# You can set min_pixels and max_pixels according to your needs, such as a token range of 256-1280, to balance performance and cost.
# min_pixels = 256*28*28
# max_pixels = 1280*28*28
# processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct", min_pixels=min_pixels, max_pixels=max_pixels)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "/home/bongo/porter_notebook/llm/qwen/samsung_20242Q.jpg",
            },
            {"type": "text", "text": "한국어 글자들 추출해서 json으로 만들어줘"},
        ],
    }
]

# Preparation for inference
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")

# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=1024)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

그리고 결과물은!!!?

['```json\n{\n  "회사의 개요": {\n    "1. 회사의 개요": {\n      "가. 회사의 법적·상업적 명칭": {\n        "당사의 명칭은 삼성전자주식회사이고 영문명은 Samsung Electronics Co., Ltd입니다."\n      },\n      "나. 설립일자": {\n        "당사는 1969년 1월 13일에 삼성전자공업주식회사로 설립되었으며, 1975년 6월 11일 기업공개를 실시하였습니다."\n      },\n      "다. 본사의 주소, 전화번호 및 홈페이지": {\n        "주소": "경기도 수원시 영통구 삼성로 129(매탄동)",\n        "전화번호": "031-200-1114",\n        "홈페이지": "https://www.samsung.com/sec"\n      }\n    },\n    "라. 주요 사업의 내용": {\n      "당사는 제품의 특성에 따라 DX(Device eXperience), DS(Device Solutions) 2개의 부문과 패널 사업을 영위하는 SDC(삼성디스플레이) 및 그 종속기업, 전장부품사업 등을 영위하는 Harman(Harman International Industries, Inc. 및 그 종속기업)으로 나누어 경영을 하고 있습니다."\n    }\n  }\n}\n```']

 

정말정말,,,못하는게 없네요! 한글도 이렇게 잘하다니....