Jun 3, 2024 · Llama3-70B: On par with GPT-3.5-Turbo; very impressive for a small model. Gemini-1.0-Pro : Bad across the board. Conclusion : I'm super impressed with the relative performance of Llama3-70B.

We train on 8xH100 GPUs with Deepspeed Zero Stage 3. We use the EasyContext implementation of Needle-in-a-Haystack to evaluate Llama-3-Giraffe-70B. We evaluate with the following parameters: We’re on a journey to advance and democratize artificial intelligence through open source and open science.

May 17, 2024 · Correction: all Llama-3-instruct finetunes are performing worse that the instruct model. Finetunes based on Llama-3 base model can outperform Meta instruct finetune. Try https://huggingface.co/bartowski/SFR-Iterative-DPO-LLaMA-3-8B-R-GGUF or Hermes-2-Pro-Llama-3-8B-GGUF-Q8_0.

May 3, 2024 · Here's another fun experiment from me! Earlier I created a simple Python script to perform a single-needle-in-a-haystack test on Ollama. I was shocked that llama3-gradient:8b-instruct-1048k-q8_0 perfectly recalled the needle in 100 out of 100 tests!

Dual chunk attention is a training-free and effective method for extending the context window of large language models (LLMs) to more than 8x times their original pre-training length. We refer to the Llama-based model with dual chunk attention as ChunkLlama.

Abacus.AI presents our longer-necked variant of Llama 3 70B - now with the instruct variant! This model has an effective context length of approximately 128k. We have currently trained on ~1.5B tokens. There are our Needle-in-a-Haystack heatmap results.

We develop Task Haystack, which comprises 64 classification tasks, to assess and diagnose how long-context LMs utilize contexts in Lifelong ICL We benchmark 14 models and find that (1) SOTA model (GPT-4o) fails ~11% cases (2) Llama-3.1-70b shows the best performance among open-weight models (3) Other open-weight models lag behind by a large margin

Llama-3-Taiwan-70B is a 70B parameter model finetuned on a large corpus of Traditional Mandarin and English data using the Llama-3 architecture. It demonstrates state-of-the-art performance on various Traditional Mandarin NLP benchmarks.

We are actively testing DCA with Llama3 8B/70B. The results will be updated next week! Hi! The language modeling results for Llama3 8b/70b have been updated! DCA is still very effective! ChunkLlama3 shows an obvious performance gain over ChunkLlama2 in PPL. Updates of needle in a haystack, few-shot, and zero-shot results are expected in two days.

Q2_K is the first quantization of Llama 3 70B that didn't achieve a perfect score in the regular runs. Therefore, I recommend using at least a 3-bit, or ideally a 4-bit, quantization of the 70B. However, even at Q2_K, the 70B remains a better choice than the unquantized 8B.