Transformers documentation
EXAONE
This model was contributed to Hugging Face Transformers on 2026-11-19.
EXAONE
EXAONE covers the EXAONE 3.0, EXAONE 3.5 and EXAONE Deep language models developed by LG AI Research. Their architecture is the same as Llama, with different parameter and config names.
The auto classes load them with the Llama implementation (LlamaModel and LlamaForCausalLM), so only ExaoneConfig is specific to EXAONE. The released checkpoints are renamed to the Llama names when loading and back when saving.
The example below demonstrates how to generate text with the AutoModelForCausalLM class.
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct")
model = AutoModelForCausalLM.from_pretrained("LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct", device_map="auto")
messages = [{"role": "user", "content": "Explain the Pythagorean theorem in one sentence."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)
output = model.generate(**inputs.to(model.device), max_new_tokens=64)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1] :], skip_special_tokens=True))ExaoneConfig
class transformers.ExaoneConfig
< source >( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: str | torch.dtype | None = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: Literal['regression', 'single_label_classification', 'multi_label_classification'] | None = Nonevocab_size: int = 102400hidden_size: int = 4096intermediate_size: int = 14336num_hidden_layers: int = 32num_attention_heads: int = 32num_key_value_heads: int | None = 8hidden_act: str = 'silu'max_position_embeddings: int = 4096initializer_range: float = 0.02rms_norm_eps: float = 1e-05use_cache: bool = Truepad_token_id: int | None = 0bos_token_id: int | None = 1eos_token_id: int | list[int] | None = 361tie_word_embeddings: bool = Falserope_parameters: transformers.modeling_rope_utils.RopeParameters | dict | None = Noneattention_bias: bool = Falseattention_dropout: int | float | None = 0.0mlp_bias: bool = Falsehead_dim: int | None = None )
Parameters
- vocab_size (
int, optional, defaults to102400) — Vocabulary size of the model. Defines the number of different tokens that can be represented by theinput_ids. - hidden_size (
int, optional, defaults to4096) — Dimension of the hidden representations. - intermediate_size (
int, optional, defaults to14336) — Dimension of the MLP representations. - num_hidden_layers (
int, optional, defaults to32) — Number of hidden layers in the Transformer decoder. - num_attention_heads (
int, optional, defaults to32) — Number of attention heads for each attention layer in the Transformer decoder. - num_key_value_heads (
int, optional, defaults to8) — This is the number of key_value heads that should be used to implement Grouped Query Attention. Ifnum_key_value_heads=num_attention_heads, the model will use Multi Head Attention (MHA), ifnum_key_value_heads=1the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out this paper. If it is not specified, will default tonum_attention_heads. - hidden_act (
str, optional, defaults tosilu) — The non-linear activation function (function or string) in the decoder. For example,"gelu","relu","silu", etc. - max_position_embeddings (
int, optional, defaults to4096) — The maximum sequence length that this model might ever be used with. - initializer_range (
float, optional, defaults to0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices. - rms_norm_eps (
float, optional, defaults to1e-05) — The epsilon used by the rms normalization layers. - use_cache (
bool, optional, defaults toTrue) — Whether or not the model should return the last key/values attentions (not used by all models). Only relevant ifconfig.is_decoder=Trueor when the model is a decoder-only generative model. - pad_token_id (
int, optional, defaults to0) — Token id used for padding in the vocabulary. - bos_token_id (
int, optional, defaults to1) — Token id used for beginning-of-stream in the vocabulary. - eos_token_id (
Union[int, list[int]], optional, defaults to361) — Token id used for end-of-stream in the vocabulary. - tie_word_embeddings (
bool, optional, defaults toFalse) — Whether to tie weight embeddings according to model’stied_weights_keysmapping. - rope_parameters (
Union[~modeling_rope_utils.RopeParameters, dict], optional) — Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value forrope_thetaand optionally parameters used for scaling in case you want to use RoPE with longermax_position_embeddings. - attention_bias (
bool, optional, defaults toFalse) — Whether to use a bias in the query, key, value and output projection layers during self-attention. - attention_dropout (
Union[int, float], optional, defaults to0.0) — The dropout ratio for the attention probabilities. - mlp_bias (
bool, optional, defaults toFalse) — Whether to use a bias in up_proj, down_proj and gate_proj layers in the MLP layers. - head_dim (
int, optional) — The attention head dimension. If None, it will default to hidden_size // num_attention_heads
This is the configuration class to store the configuration of a LlamaModel. It is used to instantiate a Exaone model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct
Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.
>>> from transformers import AutoModelForCausalLM, ExaoneConfig
>>> # Initializing an EXAONE-3.0-7.8B style configuration
>>> configuration = ExaoneConfig()
>>> # Initializing a model (which uses the Llama implementation) from that configuration
>>> model = AutoModelForCausalLM.from_config(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config