llama-recipes/recipes

• Running Llama2 in Google Colab

This folder contains examples organized by topic:

Subfolder	Description
quickstart	The "Hello World" of using Llama2, start here if you are new to using Llama2
multilingual	Scripts to add a new language to Llama2
finetuning	Scripts to finetune Llama2 on single-GPU and multi-GPU setups
inference	Scripts to deploy Llama2 for inference locally and using model servers
use_cases	Scripts showing common applications of Llama2
responsible_ai	Scripts to use PurpleLlama for safeguarding model outputs
llama_api_providers	Scripts to run inference on Llama via hosted endpoints
benchmarks	Scripts to benchmark Llama 2 models inference on various backends
code_llama	Scripts to run inference with the Code Llama models
evaluation	Scripts to evaluate fine-tuned Llama2 models using lm-evaluation-harness from EleutherAI

Note on using Replicate
To run some of the demo apps here, you'll need to first sign in with Replicate with your github account, then create a free API token here that you can use for a while. After the free trial ends, you'll need to enter billing info to continue to use Llama2 hosted on Replicate - according to Replicate's Run time and cost for the Llama2-13b-chat model used in our demo apps, the model "costs $0.000725 per second. Predictions typically complete within 10 seconds." This means each call to the Llama2-13b-chat model costs less than $0.01 if the call completes within 10 seconds. If you want absolutely no costs, you can refer to the section "Running Llama2 locally on Mac" above or the "Running Llama2 in Google Colab" below.

Note on using OctoAI
You can also use OctoAI to run some of the Llama demos under OctoAI_API_examples. You can sign into OctoAI with your Google or GitHub account, which will give you $10 of free credits you can use for a month. Llama2 on OctoAI is priced at $0.00086 per 1k tokens (a ~350-word LLM response), so $10 of free credits should go a very long way (about 10,000 LLM inferences).

Running Llama2 in Google Colab

To run Llama2 in Google Colab using llama-cpp-python, download the quantized Llama2-7b-chat model here, or follow the instructions above to build it, before uploading it to your Google drive. Note that on the free Colab T4 GPU, the call to Llama could take more than 20 minutes to return; running the notebook locally on M1 MBP takes about 20 seconds.