• CXR LLaVA
• • • Multimodal Large Language Model Fine-Tuned for Chest X-ray Images
  • Usage Guide
  • • Install Dependencies
    • Importing Packages
    • Prepare CXR
    • Load model
    • Generating Radiologic Reports
    • Differential Diagnosis
    • Question Answering
    • Custom Prompt
  • License Information

CXR LLaVA

Multimodal Large Language Model Fine-Tuned for Chest X-ray Images

CXR LLaVA is an innovative open-source, multimodal large language model specifically designed for generating radiologic reports from chest X-ray images.

• Arxiv Preprint Paper: Explore the detailed scientific background of CXR LLaVA on Arxiv.
• Demo Website: Experience the model in action at Radiologist App.

Version	Input CXR resolution	Channels	Vision Encoder	Base LLM	Weight
v1.0	512x512	RGB	RN50	LLAMA2-13B-CHAT	Deprecated
v2.0.1 (Latest)	512x512	Grayscale	ViT-L/16	LLAMA2-7B-CHAT	Link

You can interpret CXR with just 6 lines of code.

(NVIDIA GPU VRAM>14GB needed)

from transformers import AutoModel
from PIL import Image
model = AutoModel.from_pretrained("ECOFRI/CXR-LLAVA-v2", trust_remote_code=True)
model = model.to("cuda")
cxr_image = Image.open("img.jpg")
response = model.write_radiologic_report(cxr_image)

The radiologic report reveals a large consolidation in the right upper lobe of the lungs. There is no evidence of pleural effusion or pneumothorax. The cardiac and mediastinal contours are normal.

Usage Guide

Install Dependencies

Before you begin, make sure you have PyTorch installed. After confirming that PyTorch is installed, you can install the additional required dependencies. Run the following command in your terminal or command prompt:

pip install transformers sentencepiece protobuf pillow

Importing Packages

from transformers import AutoModel
from PIL import Image

Prepare CXR

Ensure you have an CXR image file ready, such as 'img.jpg'.

Use the following code to load the image

cxr_image = Image.open("img.jpg")

Load model

Loading the CXR-LLAVA model is straightforward and can be done in one line of code.

model = AutoModel.from_pretrained("ECOFRI/CXR-LLAVA-v2", trust_remote_code=True)
model = model.to("cuda")

Generating Radiologic Reports

To write a radiologic report of a chest radiograph:

response = model.write_radiologic_report(cxr_image)

The radiologic report reveals a large consolidation in the right upper lobe of the lungs. There is no evidence of pleural effusion or pneumothorax. The cardiac and mediastinal contours are normal.

Differential Diagnosis

For differential diagnosis:

response = model.write_differential_diagnosis(cxr_image)

Possible differential diagnoses for this patient include pneumonia,tuberculosis, lung abscess, or a neoplastic process such as lung cancer.

Question Answering

To ask a question:

question = "What is true meaning of consolidation?"
response = model.ask_question(question=question, image=cxr_image)

Consolidation refers to the filling of the airspaces in the lungs with fluid, pus, blood, cells or other substances, resulting in a region of lung tissue that has become dense and solid rather than containing air.

Custom Prompt

For custom interactions:

img = Image.open("img.jpg")
chat = [
    {"role": "system",
     "content": "You are a helpful radiologist. Try to interpret chest x ray image and answer to the question that user provides."},
    {"role": "user",
     "content": "<image>\nWrite a radiologic report on the given chest radiograph, including information about atelectasis, cardiomegaly, consolidation, pulmonary edema, pleural effusion, and pneumothorax.\n"}
]
response = model.generate_cxr_repsonse(chat=chat,pil_image=img, temperature=0, top_p=1)

License Information

CXR LLaVA is available under a Creative Commons NonCommercial License.

Users must obtain the LLAMA-2 license prior to use. More details can be found here.

Lastly, we extend our heartfelt thanks to all the contributors of the LLaVA project.