jlxddlZddlmZddlZddlmZddlm Z m Z ej e Z GddZdS)N) __version__) DynamicCacheEncoderDecoderCachec(eZdZdZedZeddefdZedZedZ ede e e j fd Z edd e e j d efd Zedde e e j fdZdS)PastKeyValuesHelperzEHelper functions to process past key values for encoder-decoder modelct|dkr|d|d|d|dfSt|dksJdt||S)Nrz(Expected to have four or six items. Got )len) present_layers /Users/tedpeters/Documents/Projects/Claude Projects/Misc Tools/notename-sheets/.venv/lib/python3.11/site-packages/onnxruntime/transformers/past_helper.py _unpack_layerz!PastKeyValuesHelper._unpack_layerss }   " " #]1%5}Q7GWXIYY Y=!!Q&&&(gSVWdSeSe(g(g&&&Fpresentcg}g}t|D]T}||r d|d|gn d|d|g||r d|d|gn d|d|gU||zS) Npresent_key_self_present_value_self_past_key_self_past_value_self_present_key_cross_present_value_cross_past_key_cross_past_value_cross_)rangeextend) num_layersrpast_self_namespast_cross_namesis rget_past_namesz"PastKeyValuesHelper.get_past_namessz""  A  " "D(Q((*C*C*CDD*q**,Bq,B,BC     # #F)a))+E!+E+EFF+++-D-D-DE     !111rcg}g}t|D]R\}}t|\}}}}|||g|||gS||fS)aSplit present state from grouped by layer to grouped by self/cross attention. Before: (past_key_self_0, past_value_self_0, past_key_cross_0, past_value_cross_0), (past_key_self_1, past_value_self_1, past_key_cross_1, past_value_cross_1), ... After: (past_key_self_0, past_value_self_0, past_key_self_1, past_value_self_1, ...), (past_key_cross_0, past_value_cross_0, past_key_cross_1, past_value_cross_1, ...)  enumeraterrr) present_key_values present_self present_cross_ipresent_layer_ipresent_key_selfpresent_value_selfpresent_key_crosspresent_value_crosss rgroup_by_self_or_crossz*PastKeyValuesHelper.group_by_self_or_cross,s  #,-?#@#@ K K B $11/BB   "!#   !13E F G G G  "35H!I J J J J]**rcntdzksJtfdtD}tjt tjdkrBt d|D}t d|D}t||S|S)aReorder past state from grouped by self/cross attention to grouped by layer. Before: past_key_self_0, past_value_self_0, past_key_self_1, past_value_self_1, ..., past_key_cross_0, past_value_cross_0, past_key_cross_1, past_value_cross_1, ... After: (past_key_self_0, past_value_self_0, past_key_cross_0, past_value_cross_0), (past_key_self_1, past_value_self_1, past_key_cross_1, past_value_cross_1), r c3K|]@}d|zd|zdzdzd|zzdzd|zzdzgVAdS)r N).0r!rpasts r z5PastKeyValuesHelper.group_by_layer..Gs   QU QUQYQ^a!e+,Q^a!e+a/0        rz5.0c36K|]}|d|dfVdS)rr Nr3r4layers rr6z5PastKeyValuesHelper.group_by_layer..Qs//Y/Yq580D/Y/Y/Y/Y/Y/Yrc36K|]}|d|dfVdS)r2r Nr3r8s rr6z5PastKeyValuesHelper.group_by_layer..Rs/0Z0Z%%(E!H1E0Z0Z0Z0Z0Z0Zr)r tuplerpvVersiontransformers_versionrr)r5rlayersself_attention_cachecross_attention_caches`` rgroup_by_layerz"PastKeyValuesHelper.group_by_layer@s 4yyA N****     :&&      :* + +rz%/@/@ @ @#//Y/YRX/Y/Y/Y#Y#Y $00Z0ZSY0Z0Z0Z$Z$Z !&';=RSS S rpast_key_valuescd}t|dz}tt|dzD]5}d|z}|||||dz|||z|||zdzffz }6|S)aCategorize present_key_values from self and cross attention to layer by layer. Reorder past state from grouped by self/cross attention to grouped by layer. Before: past_key_self_0, past_value_self_0, past_key_self_1, past_value_self_1, ..., past_key_cross_0, past_value_cross_0, past_key_cross_1, past_value_cross_1, ... After: (past_key_self_0, past_value_self_0, past_key_cross_0, past_value_cross_0), (past_key_self_1, past_value_self_1, past_key_cross_1, past_value_cross_1), Args: present_key_values: From past_key_values of a model (group by self and cross attention) Returns: past_tuples: present key and values grouped by layer. r3r2r r )r r)rC past_tupleshalf_idxr!idxs rback_group_by_layerz'PastKeyValuesHelper.back_group_by_layerVs  ''1,s?++q011  Aa%C #C(#C!G,#HsN3#HsNQ$67  KKrr&concatcg}g}t|D]R\}}t|\}}}} |||g||| gS|r||zS||fS)aCategorize present_key_values into self and cross attention. Split present state from grouped by layer to grouped by self/cross attention. Before: (past_key_self_0, past_value_self_0, past_key_cross_0, past_value_cross_0), (past_key_self_1, past_value_self_1, past_key_cross_1, past_value_cross_1), ... After: (past_key_self_0, past_value_self_0, past_key_self_1, past_value_self_1, ...), (past_key_cross_0, past_value_cross_0, past_key_cross_1, past_value_cross_1, ...) Args: present_key_values: From past_key_values of a model (group by layer) concat: If concat self attention with cross attention key/value to return Returns: present_self (Tuple[torch.Tensor]): present key and values from self attention present_cross (Tuple[torch.Tensor]): present key and values from cross attention r$) r&rIr'r(_r*r+r,r-r.s rgroup_by_self_and_crossz+PastKeyValuesHelper.group_by_self_and_crossts$,. ,. "+,>"?"? K K A#11/BB Y 02CEX   !13E F G G G  "35H!I J J J J  /-/ /. .rTcJg}|rt|dznt|}|sdndt|D]+}|fdd|d|fD,t|D]+}|fdd|d |fD,|S) zProcess input names of model wrapper. Args: past_key_values: Consider `self` and `cross` past_key_values Returns: names (List[string]): input names r past_present_cg|]}|zSr3r3r4sprefixs r z7PastKeyValuesHelper.get_input_names..sSSS&1*SSSr key_self_ value_self_cg|]}|zSr3r3rQs rrTz7PastKeyValuesHelper.get_input_names..sUUU&1*UUUr key_cross_ value_cross_)r rr)rCencodernamesrr!rSs @rget_input_namesz#PastKeyValuesHelper.get_input_namess29SS))Q..s??S?S '7Zz"" U UA LLSSSS1@Qa@Q@Q.RSSS T T T Tz"" W WA LLUUUU/?A/?/?ASPQASAS.TUUU V V V V rN)F)T)__name__ __module__ __qualname____doc__ staticmethodrboolr"r/rBr;torchTensorrHrLr\r3rrrrs;OO\ 22D222\2 ++\+&\*U53F-G\://E%,4G/QU///\/<uU\/B)C\rr)loggingpackaging.versionversionr<rc transformersrr>transformers.cache_utilsrr getLoggerr]loggerrr3rrrls <<<<<<FFFFFFFF  8 $ $SSSSSSSSSSr