MemeTT

Official repository for the ICWSM '26 paper, "Large Scale Narrative Analysis of Multimodal Memes."

Datasets

• Harm-C

Pramanick, S.; Dimitrov, D.; Mukherjee, R.; Sharma, S.; Akhtar, M. S.; Nakov, P.; and Chakraborty, T. 2021a. Detecting Harmful Memes and Their Targets. In Zong, C.; Xia, F.; Li, W.; and Navigli, R., eds., Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 2783–2796. Online: Association for Computational Linguistics.

• Harm-P

Pramanick, S.; Sharma, S.; Dimitrov, D.; Akhtar, M. S.; Nakov, P.; and Chakraborty, T. 2021b. MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets. In Moens, M.-F.; Huang, X.; Specia, L.; and Yih, S. W.-t., eds., Findings of the Association for Computational Linguistics: EMNLP 2021, 4439–4455. Punta Cana, Dominican Republic: Association for Computational Linguistics.

• TotalDefMeme

Prakash, N.; Hee, M. S.; and Lee, R. K.-W. 2023. TotalDefMeme: A Multi-Attribute Meme dataset on Total Defence in Singapore. In Proceedings of the 14th ACM Multimedia Systems Conference, MMSys ’23, 369–375. New York, NY, USA: Association for Computing Machinery. ISBN 9798400701481.

Note: All datasets are publicly available.

---

Usage

Step 1: Prepare Dependencies and Credentials

1. Create a new teamspace on Lightning AI, following the guide to create a teamspace.

2. Download all files from the uploads folder in this repository. Drag-and-drop them into the uploads folder on your Lightning AI teamspace's Drive.

3. a. Create an access key pair for an IAM user via the AWS Identity and Access Management (IAM) console, following this guide.

   b. Save the key pair in a .env file named AWS_CREDENTIALS.env, in the following format:

   AWS_ACCESS_KEY_ID=<insert AWS access key>
   AWS_SECRET_ACCESS_KEY=<insert AWS secret key>
   AWS_DEFAULT_REGION=<insert AWS Region>
   AWS_MAX_ATTEMPTS=10
   AWS_RETRY_MODE="standard"

   c. Upload the AWS_CREDENTIALS.env file to the uploads folder on your Lightning AI teamspace's Drive.

4. a. Create a Google Cloud service account, following the guide to creating a service account and the guide to creating a service account key. Choose JSON as your key type. Rename the JSON file as GOOGLE_APPLICATION_CREDENTIALS.json.

   b. Enable the Cloud Vision API under APIs & Services in your Google Cloud Console.

   c. Upload the GOOGLE_APPLICATION_CREDENTIALS.json file to the uploads folder on your Lightning AI teamspace's Drive.

5. a. Create a Google Cloud service account, following the guide to creating a service account and the guide to creating a service account key. Choose JSON as your key type. Rename the JSON file as GOOGLE_APPLICATION_CREDENTIALS_VERTEX.json.

   b. Assign the Agent Platform User role to the service account created in Step 5a.

   c. Enable the Agent Platform API under APIs & Services in your Google Cloud Console.

   d. Upload the GOOGLE_APPLICATION_CREDENTIALS_VERTEX.json file to the uploads folder on your Lightning AI teamspace's Drive.

6. a. Create a .env file named GOOGLE_CLOUD_PROJECT_DETAILS.env, in the following format:

   GOOGLE_CLOUD_PROJECT=<insert Google Cloud Project ID>
   GOOGLE_CLOUD_LOCATION=<insert location>
   GOOGLE_GENAI_USE_VERTEXAI="True"

   b. Upload the GOOGLE_CLOUD_PROJECT_DETAILS.env file to the uploads folder on your Lightning AI teamspace's Drive.

7. a. Create a Fireworks AI API key following the guide, and save the key in a .env file named FIREWORKS_CREDENTIALS.env, in the following format:

   FIREWORKS_API_KEY=<insert your actual API key here>

   b. Upload the FIREWORKS_CREDENTIALS.env file to the uploads folder on your Lightning AI teamspace's Drive.

8. a. Create a Mistral AI API key following the guide, and save the key in a .env file named MISTRAL_CREDENTIALS.env, in the following format:

   MISTRAL_API_KEY=<insert your actual API key here>

   b. Upload the MISTRAL_CREDENTIALS.env file to the uploads folder on your Lightning AI teamspace's Drive.

9. a. Create a Cohere API key following the guide, and save the key in a .env file named COHERE_CREDENTIALS.env, in the following format:

   COHERE_API_KEY=<insert your actual API key here>

   b. Upload the COHERE_CREDENTIALS.env file to the uploads folder on your Lightning AI teamspace's Drive.

10. a. Create a Voyage AI API key following the guide, and save the key in a .env file named VOYAGE_CREDENTIALS.env, in the following format:

    VOYAGE_API_KEY=<insert your actual API key here>

    b. Upload the VOYAGE_CREDENTIALS.env file to the uploads folder on your Lightning AI teamspace's Drive.

11. a. Create a OpenAI API key following the guide, and save the key in a .env file named OPENAI_CREDENTIALS.env, in the following format:

    OPENAI_API_KEY=<insert your actual API key here>

    b. Upload the OPENAI_CREDENTIALS.env file to the uploads folder on your Lightning AI teamspace's Drive.

Step 2: Run Studios

Note: As existing models become deprecated, you will need to update the model names accordingly in the scripts.

Each stage in the pipeline follows the same structure:

1. For each Python script directly in the stage's folder, create a new Studio in Lightning AI using the script's filename (without .py) as the Studio name, and upload the corresponding script to it.

2. Download all the Python scripts from the stage's scripts subfolder in this repository. These scripts enable you to programmatically launch the Studios created in Step 1.

3. In a separate new Lightning AI Studio, upload and run these scripts.

Apply the three above steps to each of the following stages, in this order:

1. meme_preprocessing
2. Each subfolder under mabsa/ (12 model folders)
3. extract_quintuplets
4. embeddings
5. clustering
6. Each subfolder under llm_judge/

Note: Upload the files under the samples/rq1 and the samples/rq4 folders to the 6000_llm_judge_evaluate_quadruple_accuracy and the 6003_llm_judge_evaluate_intercluster_quality Lightning Studios, respectively, before running.