Google does not open source PaLM, but netizens open source it! Miniature version of hundreds of billions of parameters: the maximum is only 1 billion, 8k context

PaLM, which Google has not open sourced, has been open sourced by netizens.

Yesterday, a developer open sourced three miniature versions of the PaLM model on GitHub: parameters are 150 million (PalM-150m), 410 million (PalM-410m) and 1 billion (PalM-1b).

## Project address: https://github.com/conceptofmind/PaLM

These three models were trained on the Google C4 data set with a context length of 8k. In the future, models with 2 billion parameters are being trained.

Google C4 Dataset

Example generated using 410 million parameter model:

Although the parameters are indeed a bit small, the generated effect is still a bit hard to describe...

These models are compatible with many popular Lucidrain repositories, such as Toolformer-pytorch , PalM-rlhf-pytorch and PalM-pytorch.

The three latest open source models are baseline models and will be trained on larger-scale data sets.

All models will be further adjusted with instructions on FLAN to provide flan-PaLM models.

In terms of optimization algorithm, the decoupled weight attenuation Adam W is used, but you can also choose to use Mitchell Wortsman's Stable Adam W.

Currently, the model has been uploaded to Torch hub and the files are also stored in Huggingface hub.

If the model cannot be downloaded correctly from Torch hub, be sure to clear the checkpoints and model folders in .cache/torch/hub/. If the issue is still not resolved, then you can download the file from Huggingface’s repository. Currently, the integration of Huggingface is in progress.

All training data has been pre-labeled with GPTNEOX tagger, and the sequence length is cut off to 8192. This will help save significant costs in preprocessing data.

These datasets have been stored on Huggingface in parquet format, you can find the individual data chunks here: C4 Chunk 1, C4 Chunk 2, C4 Chunk 3, C4 Chunk 4, and C4 Chunk 5.

There is another option in the distributed training script to not use the provided pre-labeled C4 dataset, but to load and process another dataset like openwebtext.

Installation

There is a wave of installation required before attempting to run the model.

git clone https://github.com/conceptofmind/PaLM.gitcd PaLM/pip3 install -r requirements.txt

Using

You can do additional training or fine-tuning by loading a pre-trained model using Torch hub:

model = torch.hub.load("conceptofmind/PaLM", "palm_410m_8k_v0").cuda()

In addition, you can also directly load the PyTorch model checkpoint through the following method:

from palm_rlhf_pytorch import PaLMmodel = PaLM(num_tokens=50304, dim=1024, depth=24, dim_head=128, heads=8, flash_attn=True, qk_rmsnorm = False,).cuda()model.load('/palm_410m_8k_v0.pt')

To use the model to generate text, you can use the command line:

prompt - Prompt for generating text.

seq _ len - the sequence length of the generated text, the default value is 256.

temperature - Sampling temperature, default is 0.8

filter_thres - Filter threshold used for sampling. The default value is 0.9.

model - The model to use for generation. There are three different parameters (150m, 410m, 1b): palm_150m_8k_v0, palm_410m_8k_v0, palm_1b_8k_v0.

python3 inference.py "My dog is very cute" --seq_len 256 --temperature 0.8 --filter_thres 0.9 --model "palm_410m_8k_v0"

In order to improve performance, reasoning uses torch.compile(), Flash Attention and Hidet.

If you want to extend the generation by adding stream processing or other functions, the author provides a general inference script "inference.py".

Training

These "Open Source Palm" models were trained on 64 A100 (80GB) GPUs.

In order to facilitate model training, the author also provides a distributed training script train_distributed.py.

You are free to change the model layer and hyperparameter configuration to meet the hardware requirements, and you can also load the model's weights and change the training script to fine-tune the model.

Finally, the author stated that he would add a specific fine-tuning script and explore LoRA in the future.

Data

can be generated by running the build_dataset.py script similar to C4 used during training Dataset way to preprocess different datasets. This will pre-label the data, split it into chunks of specified sequence length, and upload it to the Huggingface hub.

For example:

python3 build_dataset.py --seed 42 --seq_len 8192 --hf_account "your_hf_account" --tokenizer "EleutherAI/gpt-neox-20b" --dataset_name "EleutherAI/the_pile_deduplicated"

PaLM 2 is coming

In April 2022, Google officially announced 540 billion for the first time Parameters of PaLM. Like other LLMs, PaLM can perform a variety of text generation and editing tasks.

PaLM is Google’s first large-scale use of the Pathways system to scale training to 6144 chips, which is the largest TPU-based system configuration used for training to date.

Its understanding ability is outstanding. Not only can it understand the jokes, but it can also explain the funny points to you who don’t understand.

Just in mid-March, Google opened its PaLM large language model API for the first time.

This means that people can use it to complete tasks such as summarizing text, writing code, and even training PaLM into a ChatGPT-like Conversational chatbot.

At the upcoming Google annual I/O conference, Pichai will announce the company’s latest developments in the field of AI.

It is said that the latest and most advanced large-scale language model PaLM 2 will be launched soon.

PaLM 2 includes more than 100 languages ​​and has been running under the internal code name "Unified Language Model" (Unified Language Model). It also conducts extensive testing in coding and mathematics as well as creative writing.

Last month, Google said that its medical LLM "Med-PalM2" can answer medical exam questions with an accuracy of 85% at the "expert doctor level."

In addition, Google will also release Bard, a chat robot supported by large models, as well as a generative experience for search.

It remains to be seen whether the latest AI release can straighten Google’s back.

The above is the detailed content of Google does not open source PaLM, but netizens open source it! Miniature version of hundreds of billions of parameters: the maximum is only 1 billion, 8k context. For more information, please follow other related articles on the PHP Chinese website!