\j"0dZddlmZmZmZddlZddlmZm Z m Z Gdde jZ Gdde jZ Gd d e jZGd d e jZGd de jZGdde jZGdde j$ZGdde j(ZGdde j,ZGdde j0ZGdde j,ZGdde j0Zy)z0Declares specification of the Transformer model.)OptionalTupleUnionN)attention_spec common_spec model_specc/LeZdZddejj dej jddddddddddddddddfdedede d e d ejd ed ej d e de de de de de de ede ede ede de e jde de de ede e de f.dZy)TransformerEncoderSpecTFN' num_layers num_headspre_norm no_final_norm activationnum_source_embeddingsembeddings_mergelayernorm_embeddingrelative_positionrelative_attention_biasffn_glurms_normmulti_query_attention num_heads_kvhead_dim rotary_dimrotary_interleaverotary_scaling_typerotary_scaling_factor rotary_basesliding_windowqk_normpre_post_layer_normc"| r||dk7r tdd}| |_tjdj ||_||_tjdj ||_tjdj ||_t|Dcgc]}tjc}|_ d|_ | s| st|_|r|stj | |_|rtj | |_|)tjdj ||_t|Dcgc]}t)| | | | |||||||||| c}|_ycc}wcc}w) a'Initializes a Transformer encoder specification. Args: num_layers: Number of layers. num_heads: Number of attention heads. pre_norm: Enable the pre-norm Transformer architecture. no_final_norm: Disable the final layer norm in the pre-norm architecture. activation: Activation to apply in the feed-forward network. num_source_embeddings: Number of source embeddings. embeddings_merge: When :obj:`num_source_embeddings` > 1, specify how the embeddings are merged. layernorm_embedding: Apply layer normalization after the embedding layer. relative_position: Use relative position representations in the self-attention layers as described in https://arxiv.org/abs/1803.02155. relative_attention_bias: Use relative attention bias in the self-attention layers as described in the T5 paper https://arxiv.org/abs/1910.10683. ffn_glu: Use gated linear units in the FFN layers as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. multi_query_attention: Use multi-query attention (alias for num_heads_kv=1). num_heads_kv: Number of attention heads for the key and value. head_dim: Number of dimensions per attention head. rotary_dim: Apply rotary embeddings to these first N dimensions. If 0, rotary embeddings are applied to all dimensions. rotary_interleave: Interleave the head dimensions when rotary embeddings are applied. Otherwise the head dimensions are sliced in half. rotary_scaling_type: Type of RoPE scaling. rotary_scaling_factor: Factor used in the RoPE scaling. rotary_base: The base period of the rotary embeddings. sliding_window: Max sequence length to retain in KV Cache. qk_norm: Apply layer normalization to the query and key projections. pre_post_layer_norm: Add post layer norm for each pre norm layer. Nr 5Enabling multi_query_attention implies num_heads_kv=1int16int8Trint32)rrrrrrr!rrrrr r"r#) ValueErrorrnpdtypetyperrrrrangerEmbeddingsSpec embeddingsscale_embeddingsPositionEncoderSpecposition_encodings LayerNormSpec layer_normrr!TransformerEncoderLayerSpeclayer)selfr rrrrrrrrrrrrrrrrrrr r!r"r#_s U/root/.hermes/venv/lib/python3.12/site-packages/ctranslate2/specs/transformer_spec.py__init__zTransformerEncoderSpec.__init__ szx !'LA,= KL%:"'*// :  ((6*// ; " 0 5 56F G278M2N -.K & & ( !% )@&9&;D # M)77JDO '2'@'@('SD $  %"$((7"3"8"8"HD &:&# "! ("3(?!)!-%"3$7&;'$7     s -F F )__name__ __module__ __qualname__r ActivationRELUEmbeddingsMergeCONCATintboolrrRotaryScalingTypefloatr;r:r r s #-8-C-C-H-H%&8C8S8S8Z8Z$)"'(-&+&*"&$("&JN'("(,"'$)1g g g  g  g  ** g  #g &55g "g  g "&g g g  $g smg 3-!g "SM#g $ %g &&n&F&FG'g ( %)g *+g ,! -g .$/g 0"1g rHr cMeZdZdejj ddddddddddddddddddddddddddddddddddf$ded ed ed ejd ed ededededededededededededeededee jde de deded ed!ed"ed#ed$eed%eed&eed'eejd(eed)eed*ed+ed,eed-efLd.Z ed/Zy)0TransformerDecoderSpecTFr Nr rr rrrrwith_encoder_attentionrproject_in_outrralignment_layeralignment_headsrralibialibi_use_positive_positions scale_alibirrrrr original_max_position_embeddingsmax_position_embeddingsparallel_residualshared_layer_normr#rrrr! quant_typequant_group_size quant_bitsr"v_norm external_pre_post_encoder_layersmerged_encoder_attentionc' t|_|r|s td|r td|r||dk7r tdd}tjdj ||_||_tjdj ||_tjdj | |_ tjdj | |_ tj|_ d|_tj |_||_||_||_|)tjd j ||_| s| s|s|t-|_|r|stj0| |_|rtj0| |_tj6|_t;|D'cgc]M}'t=d&id |d | d | d| d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|#d|$d|%d |&Oc}'|_d!|_ tj |_!|xs||k7|jd"<|r2tj6|_"tj6|_#| r.| |jd#<|"|jd$<|!|jd%<yycc}'w)'a. Initializes a Transformer decoder specification. Args: num_layers: Number of layers. num_heads: Number of attention heads. pre_norm: Enable the pre-norm Transformer architecture. activation: Activation to apply in the feed-forward network. layernorm_embedding: Apply layer normalization after the embedding layer. with_encoder_attention: Enable the encoder attention sublayers. no_final_norm: Disable the final layer norm in the pre-norm architecture. project_in_out: Add linear transformations after the embedding layer and before the final layer. relative_position: Use relative position representations in the self-attention layers as described in https://arxiv.org/abs/1803.02155. relative_attention_bias: Use relative attention bias in the self-attention layers as described in the T5 paper https://arxiv.org/abs/1910.10683. alignment_layer: Layer index selected for alignment. alignment_heads: Number of attention heads selected for alignment. ffn_glu: Use gated linear units in the FFN layers as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. alibi: Use attention with linear biases. alibi_use_positive_positions: Use positive positions in the ALiBi definition. scale_alibi: Apply the dot product scale factor to ALiBi. rotary_dim: Apply rotary embeddings to these first N dimensions. If 0, rotary embeddings are applied to all dimensions. rotary_interleave: Interleave the head dimensions when rotary embeddings are applied. Otherwise the head dimensions are sliced in half. rotary_scaling_type: Type of RoPE scaling. rotary_scaling_factor: Factor used in the RoPE scaling. rotary_base: The base period of the rotary embeddings. original_max_position_embeddings: The original max position embeddings for Su rope embeddings max_position_embeddings: The max position embeddings for Su rope embeddings parallel_residual: Use parallel residual connections in each layer block, as used by the GPT-J and GPT-NeoX models. shared_layer_norm: When using parallel residual, share the input and post attention layer norms. pre_post_layer_norm: Add post layer norm for each pre norm layer multi_query_attention: Use multi-query attention (alias for num_heads_kv=1). num_heads_kv: Number of attention heads for the key and value. sliding_window: Max sequence length to retain in KV Cache. quant_type: quantization type used (like awq... for lower bit quantization) quant_group_size: group size of the lower bit quantization quant_bits: number of bit of the quantization (ex: 4bit) external_pre_post_encoder_layers: if the encoder attention pre and processing is done outside the attention. z/The GPT-J block expects a pre-norm architecturez-The GPT-J block does not have cross attentionNr r%r&r'Tr)r(rLrrrrrrrrr rSrTrUrVr#rrr!r"rZr[r\Frquantization_typequantization_bitsquantization_group_sizerG)$dict_configr*r+r,r-rrrrNrOrr/r0r1rOPTIONAL scale_outputsrPrQrRr!r2r3r4r5r LinearSpec projectionr.TransformerDecoderLayerSpecr7start_from_zero_embeddingfinal_logit_softcapping project_in project_out)(r8r rrrrrLrrMrrrNrOrrrPrQrRrrrrr rSrTrUrVr#rrrr!rWrXrYr"rZr[r\r9s( r:r;zTransformerDecoderSpec.__init__vstv  !RSS% !PQQ 'LA,= KL'*// :  ((6*// ;!xx055oF!xx055oF%446 $'00 ,H)&  %"$((7"3"8"8"HD !+"&9&;D # M)77JDO '2'@'@('SD $%0024:&3 21 ( '= "3 )@    "   &  #4 %8 '< ( 2R )@ #4 #4 %8  *! ""# $ .% & ' () *2R+ ,*B-   8*/&'1':':$0E1 I % ,- )446DO*557D  0:DLL, -0:DLL, -6FDLL2 3 M sAK c|jSN)rbr8s r:configzTransformerDecoderSpec.config"s ||rH)r<r=r>rr?r@rCrDrrrErF Quantizationr;propertyrorGrHr:rJrJus -8-C-C-H-H$)'+#$"'(-! -2!$("&JN'("01'("'"'$)&+&*"&(,9=*.$(;@).OjGjGjG jG  ** jG " jG!%jGjGjG jG"&jGjGjGjGjG !jG"'+#jG$%jG&SM'jG( )jG*&n&F&FG+jG, %-jG./jG0+.1jG2"%3jG4 5jG6 7jG8"9jG: $;jG<sm=jG>3-?jG@! AjGB[556CjGD#3-EjGFSMGjGHIjGJKjGL+34.MjGN#'OjGXrHrJc jeZdZ d deededeejdededef dZ y) r6Nrrrrr r#ctjd|||||||| | | | |  |_t|||_|rt j ||_t j ||_t j ||_ t j ||_ t|jdt|jdyy)NT) self_attentionrrrrrr!rrrrr r"glurr(r5) rMultiHeadAttentionSpecrtFeedForwardSpecffnrr4input_layer_normpost_attention_layer_normpre_feedforward_layer_normpost_feedforward_layer_normdelattr)r8rrrrrrr!rrrrr r"r#s r:r;z$TransformerEncoderLayerSpec.__init__(s"-CC/$;%)!/ 3"7# #wB $/$=$=x$PD !-8-F-F!.D */:.G.G!/D +0;/H/H!0D , D'' 6 DHHl + rH)FFFFNNNNTNr r FF) r<r=r>rrCrDrrErFr;rGrHr:r6r6's~ %$("&JN'("$)/,SM/, /,&n&F&FG/, %/,/,"/,rHr6c@eZdZ ddZy)rgNc tjdiddd|d|d|d|d|d|d | d | d | d | d |d|d|d|d|d||_|r$|s"tj||||||du|_t |||_| rz|rt j|_n2t j|_ t j|_ t|jdt|j d|rt j||_ t j||_ |r8|r6t j||_ t j||_ t j||_t j||_t|jdt|j dt j"|_y)NrtTrrrrrrrr rSrTrrr!r"rZr\F)rrrr!r"has_normrur5r(rG)rrwrt attentionrxryrr4rVrzr{r~*external_post_encoder_attention_layer_norm)external_pre_encoder_attention_layer_normr|r}rrc layer_scalar)r8rLrrrrrrrrr rSrTrUrVr#rrr!r"rZr[r\s r:r;z$TransformerDecoderLayerSpec.__init__[s2-CC  / %<   "  0  !4 #8 $ .N %< &  *  ! "&># ( "*B+BB!)!-9UB DN#wB  )4)B)B)D&(3(A(A(C%1<1J1J1L. D'' 6 DHHl + $/$=$=x$PD !-8-F-F!.D *&*J--x@? --x@> /:.G.G!/D +0;/H/H!0D , D'' 6 DHHl +&//rH)TFFFFNTNr r rrFFFNNNFFFFr<r=r>r;rGrHr:rgrgZsL $ % )* !!).!&/]0rHrgceZdZddZy)rxctj||_tj|_tj|_|rtj|_yy)Nr()rr4r5relinear_0linear_1linear_0_noact)r8rvrs r:r;zFeedForwardSpec.__init__sM%33XF#..0 #..0 "-"8"8":D  rHN)FFrrGrHr:rxrxs;rHrxceZdZdZy)r2c.tj|_yrm)rrc encodingsrns r:r;zPositionEncoderSpec.__init__s#,,rHNrrGrHr:r2r2s-rHr2c0eZdZdZddeeffd ZxZS)TransformerConfigz%Configuration for Transformer models.layer_norm_epsilonc (t|dd|i|y)zInitializes the configuration for Transformer models. Args: layer_norm_epsilon: The layer norm epsilon value. **kwargs: Additional configuration. rNrGsuperr;r8rkwargs __class__s r:r;zTransformerConfig.__init__ I,>I&IrHrmr<r=r>__doc__rrFr; __classcell__rs@r:rrs/J8E?JJrHrc TeZdZdZdedeffd Zedddejjdddejjdddddf d e eeeeffd ed ed ed edejdedededejdededededefdZedZedZdZdZdZxZS)TransformerSpeczDescribes a Transformer model. The specification is invariant to hidden dimensions but requires to explicitly set the number of layers and attention heads. encoderdecoderc t|ts tdt|ts tdt|||_||_|jjd|j jy)zInitializes a Transformer model specification. Args: encoder: The encoder specification. decoder: The decoder specification. 1encoder argument must be a TransformerEncoderSpec1decoder argument must be a TransformerDecoderSpecrN) isinstancer TypeErrorrJrr;rrrb add_attributer)r8rrrs r:r;zTransformerSpec.__init__sm'#9:OP P'#9:OP P    "" #T\\%G%G rHFTrKr r rwith_relative_positionrrrrNrOrrrrrrrct|ttfr|\}}n||}}t|||||| | | || | || }t |||||| || ||| || }|||S)aCreates a Transformer model specification. Args: num_layers: Number of encoder and decoder layers, or a 2-tuple if the number is different. num_heads: Number of attention heads. with_relative_position: Use relative position representations in the self-attention layers as described in https://arxiv.org/abs/1803.02155. pre_norm: Enable the pre-norm Transformer architecture. no_final_norm: Disable the final layer norm in the pre-norm architecture. activation: Activation to apply in the feed-forward network. alignment_layer: Layer index selected for alignment. alignment_heads: Number of attention heads selected for alignment. num_source_embeddings: Number of source embeddings. embeddings_merge: When :obj:`num_source_embeddings` > 1, specify how the embeddings are merged. layernorm_embedding: Apply layer normalization after the embedding layer. relative_attention_bias: Use relative attention bias in the self-attention layers as described in the T5 paper https://arxiv.org/abs/1910.10683. ffn_glu: Use gated linear units in the FFN layer as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. multi_query_attention: Use multi-query attention. ) rrrrrrrrrrr) rrrrrrrNrOrrr)rlisttupler rJ)clsr rrrrrrNrOrrrrrrrnum_encoder_layersnum_decoder_layersrrs r: from_configzTransformerSpec.from_configsV j4- 05? 2  25? 2 (  '!"7- 34$;"7  )  '! 34$;++"7  7G$$rHcy)NrrGrns r:namezTransformerSpec.nameDs rHcy)NrGrns r:revisionzTransformerSpec.revisionHrHctSrm)rrns r:get_default_configz"TransformerSpec.get_default_configLs  ""rHc|jjDcgc]}|jjdc}Scc}wNrrr0weightshape)r8specs r:get_source_vocabulary_sizez*TransformerSpec.get_source_vocabulary_sizeOs/151H1HI !!!$IIIs <c\|jjjjdSrrr0rrrns r:get_target_vocabulary_sizez*TransformerSpec.get_target_vocabulary_sizeR#||&&--33A66rH)r<r=r>rr rJr; classmethodrr?r@rArBrrCrrDrrqrrrrrrrs@r:rrsn  - 8N * (-#-8-C-C-H-H! %&8C8S8S8Z8Z$)(-&+!O%#uS#X./O%O%!% O%  O%  O% **O%O%O% #O%&55O%"O%"&O%O%O% $!O%O%b!!#J7rHrc0eZdZdZddeeffd ZxZS)TransformerDecoderModelConfigz-Configuration for Transformer decoder models.rc (t|dd|i|y)zInitializes the configuration for Transformer decoder models. Args: layer_norm_epsilon: The layer norm epsilon value. **kwargs: Additional configuration. rNrGrrs r:r;z&TransformerDecoderModelConfig.__init__YrrHrmrrs@r:rrV7J8E?JJrHrcBeZdZdZdeffd Zedejjddddddddddddddd d ddddddddddddfd e d e d e d ejde de de de de de de de de de e de de e jdedede de de de d e d!e d"e e d#e e d$e e d%e ej d&e e d'e e d(e d)e f@d*Zed+Zed,Zd-Zd.ZxZS)/TransformerDecoderModelSpecz3Describes a Transformer decoder model (e.g. GPT-2).rct|ts tdt|||_|j j jD]!\}}|jj||#y)z|Initializes a Transformer decoder model specification. Args: decoder: The decoder specification. rN) rrJrrr;rroitemsrbr)r8rkeyvaluers r:r;z$TransformerDecoderModelSpec.__init__fsh '#9:OP P  ,,--335 3JC LL & &sE 2 3rHTFNr r rr rrrrrrMrrrrPrQrRrrrrr rSrTrUrVr#rrrr!rWrXrYr"rZc!t||fid|d|d|ddd|d|d|d | d | d | d | d | d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|d | }!||!S)!a! Creates a Transformer decoder model specification. Args: num_layers: Number of decoder layers. num_heads: Number of attention heads. pre_norm: Enable the pre-norm Transformer architecture. activation: Activation to apply in the feed-forward network. layernorm_embedding: Apply layer normalization after the embedding layer. no_final_norm: Do not apply layer normalization after the last decoder block. project_in_out: Add a linear layer after the embedding layer and another one before the final output projection. with_relative_position: Enable relative position representations modules. ffn_glu: Use gated linear units in the FFN layers as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. alibi: Use attention with linear biases. alibi_use_positive_positions: Use positive positions in the ALiBi definition. scale_alibi: Apply the dot product scale factor to ALiBi. rotary_dim: Apply rotary embeddings to these first N dimensions. If 0, rotary embeddings are applied to all dimensions. rotary_interleave: Interleave the head dimensions when rotary embeddings are applied. Otherwise the head dimensions are sliced in half. rotary_scaling_type: Type of RoPE scaling. rotary_scaling_factor: Factor used in the RoPE scaling. rotary_base: The base period of the rotary embeddings. original_max_position_embeddings: The original max position embeddings for Su rope embeddings max_position_embeddings: The max position embeddings for Su rope embeddings parallel_residual: Use parallel residual connections in each layer block, as used by the GPT-J and GPT-NeoX models. shared_layer_norm: When using parallel residual, share the input and post attention layer norms. pre_post_layer_norm: add post layer norm for each pre norm layer multi_query_attention: Use multi-query attention (alias for num_heads_kv=1). num_heads_kv: Number of attention heads for the key and value. head_dim: Number of head sliding_window: max sequence length to retain KV cache quant_type: quantization type used (like awq... for lower bit quantization) quant_group_size: group size of the lower bit quantization quant_bits: number of bit of the quantization (ex: 4bit) rrrrLFrrMrrrrPrQrRrrrrr rSrTrUrVr#rrrr!rWrXrYr"rZ)rJ)"rr rrrrrrMrrrrPrQrRrrrrr rSrTrUrVr#rrrr!rWrXrYr"rZrs" r:rz'TransformerDecoderModelSpec.from_configts^Z)  " " " " !4 " $) " (" *" 5" " " " *F" $" "" 0!" "!4#" $#8%" &$'" (.N)" *%<+" ,0-" .0/" 0!41" 2#83" 4&5" 67" 8*9" :";" <.=" >"?" @A" BC" H7|rHcy)NrJrGrns r:rz TransformerDecoderModelSpec.name'rHcy)NrGrns r:rz$TransformerDecoderModelSpec.revisionrrHctSrm)rrns r:rz.TransformerDecoderModelSpec.get_default_config ,..rHc\|jjjjdSrrrns r:get_vocabulary_sizez/TransformerDecoderModelSpec.get_vocabulary_sizerrH)r<r=r>rrJr;rrr?r@rCrDrrrErFrprrqrrrrrrs@r:rrcs[= 3 6 3 -8-C-C-H-H$)#$',-2!$("&JN'("01'("'"'$)&+&*"&(,9=*.$(Cppp p  ** p " ppp!%pppp'+ppSMp  !p"&n&F&FG#p$ %%p&'p(+.)p*"%+p, -p. /p0"1p2 $3p4sm5p63-7p8! 9p:[556;p<#3-=p>SM?p@ApBCppd((/7rHrc0eZdZdZddeeffd ZxZS)TransformerEncoderModelConfigz-Configuration for Transformer encoder models.rc (t|dd|i|y)zInitializes the configuration for Transformer encoder models. Args: layer_norm_epsilon: The layer norm epsilon value. **kwargs: Additional configuration. rNrGrrs r:r;z&TransformerEncoderModelConfig.__init__rrHrmrrs@r:rrrrHrceZdZdZdej j fdededej ffd Z e dZ e dZ d Z d ZxZS) TransformerEncoderModelSpecz2Describes a Transformer encoder model (e.g. BERT).Fr pooling_layerpooling_activationcPt|ts tdt|||_|j jd|j j|rCtj|_ tjdj||_yy)zInitializes a Transformer encoder model specification. Args: encoder: The encoder specification. pooling_layer: Add the pooling layer. pooling_activation: The activation to apply after the pooling layer. rrr'N)rr rrr;rrbrrrre pooler_denser+r,r-pooler_activation)r8rrrrs r:r;z$TransformerEncoderModelSpec.__init__s'#9:OP P   "" #T\\%G%G   + 6 6 8D %'XXf%5%:%:;M%ND " rHcy)Nr rGrns r:rz TransformerEncoderModelSpec.name rrHcy)Nr rGrns r:rz$TransformerEncoderModelSpec.revision$rrHctSrm)rrns r:rz.TransformerEncoderModelSpec.get_default_config(rrHcb|jjdjjdSrrrns r:rz/TransformerEncoderModelSpec.get_vocabulary_size+s(||&&q)0066q99rH)r<r=r>rrr?Tanhr rDr;rqrrrrrrs@r:rrsw< $5@5K5K5P5P O'OO(22 O4((/:rHr)rtypingrrrnumpyr+ctranslate2.specsrrr LayerSpecr rJr6rgrxr2SequenceToSequenceModelConfigrSequenceToSequenceModelSpecrLanguageModelConfigrLanguageModelSpecrrrrGrHr:rs6))EEh Z11h VoZ11od0,*"6"60,f^0*"6"6^0B;j**;-*..- J @@ J}7j<<}7@ JJ$B$B JP7*">">P7f JJ$B$B J):*">">):rH