Update README

README.md

@@ -0,0 +1,121 @@
+---
+language:
+- en
+tags:
+- llama
+---
+
+# OpenChat: Less is More for Open-source Models
+
+OpenChat is a series of open-source language models fine-tuned on very little diverse and high-quality multi-round conversations. The [dataset](https://huggingface.co/datasets/openchat/openchat_sharegpt4_dataset) contains only ~6K GPT-4 conversations filtered from the 90K ShareGPT conversations.
+
+Generic models:
+
+ - OpenChat: based on LLaMA-13B (2048 context length)
+   - **105.7%** of ChatGPT score on Vicuna GPT-4 evaluation
+   - **80.87%** Win-rate on AlpacaEval
+   - **🚀 Only used 6K data for finetuning!!!**
+ - OpenChat-8192: based on LLaMA-13B  (extended to 8192 context length)
+   - **106.6%** of ChatGPT score on Vicuna GPT-4 evaluation
+
+Code models:
+
+ - OpenCoderPlus: based on StarCoderPlus (native 8192 context length)
+   - **102.5%** of ChatGPT score on Vicuna GPT-4 evaluation
+   - **78.70%** Win-rate on AlpacaEval
+
+**NOTE:** Please load the pretrained models using *bfloat16*
+
+## Conversation Template
+
+The conversation template **involves concatenating tokens**.
+
+Besides base model vocabulary, an end-of-turn token `<|end_of_turn|>` is added, with id `eot_token_id`.
+
+```python
+# OpenChat
+[bos_token_id] + tokenize("Human: ") + tokenize(user_question) + [eot_token_id] + tokenize("Assistant: ")
+# OpenCoder
+tokenize("User:") + tokenize(user_question) + [eot_token_id] + tokenize("Assistant:")
+```
+
+*Hint: In BPE, `tokenize(A) + tokenize(B)` does not always equals to `tokenize(A + B)`*
+
+Following is the code for generating the conversation templates:
+
+```python
+@dataclass
+class ModelConfig:
+    # Prompt
+    system: Optional[str]
+
+    role_prefix: dict
+    ai_role: str
+    eot_token: str
+    bos_token: Optional[str] = None
+
+    # Get template
+    def generate_conversation_template(self, tokenize_fn, tokenize_special_fn, message_list):
+        tokens = []
+        masks = []
+
+        # begin of sentence (bos)
+        if self.bos_token:
+            t = tokenize_special_fn(self.bos_token)
+            tokens.append(t)
+            masks.append(False)
+
+        # System
+        if self.system:
+            t = tokenize_fn(self.system) + [tokenize_special_fn(self.eot_token)]
+            tokens.extend(t)
+            masks.extend([False] * len(t))
+
+        # Messages
+        for idx, message in enumerate(message_list):
+            # Prefix
+            t = tokenize_fn(self.role_prefix[message["from"]])
+            tokens.extend(t)
+            masks.extend([False] * len(t))
+
+            # Message
+            if "value" in message:
+                t = tokenize_fn(message["value"]) + [tokenize_special_fn(self.eot_token)]
+                tokens.extend(t)
+                masks.extend([message["from"] == self.ai_role] * len(t))
+            else:
+                assert idx == len(message_list) - 1, "Empty message for completion must be on the last."
+
+        return tokens, masks
+
+
+MODEL_CONFIG_MAP = {
+    # OpenChat / OpenChat-8192
+    "openchat": ModelConfig(
+        # Prompt
+        system=None,
+
+        role_prefix={
+            "human": "Human: ",
+            "gpt": "Assistant: "
+        },
+        ai_role="gpt",
+        eot_token="<|end_of_turn|>",
+        bos_token="<s>",
+    ),
+
+    # OpenCoder / OpenCoderPlus
+    "opencoder": ModelConfig(
+        # Prompt
+        system=None,
+
+        role_prefix={
+            "human": "User:",
+            "gpt": "Assistant:"
+        },
+        ai_role="gpt",
+        eot_token="<|end_of_turn|>",
+        bos_token=None,
+    )
+}
+```