jVddlmZddlmZddlmZddlZddlZddlZddl Z ddl m Z ddl m Z ddlmZddl mZdd l mZdd l mZdd l mZdd l mZdd l mZddl mZddlmZejdZGddejZGddZGddZGddZGddZ GddejZ!GddZ"dZ#d Z$d*d"Z%d*d#Z&d*d$Z'd+d%Z(Gd&d'e j)Z*d(Z+e"eegZ,e-d)krddl Z e j.e*dSdS),) annotationsCounter)KeysViewN)common)safeToParallize) StreamType) converter)corpus) environment) exceptions21)note)stream)text) MetadataEntryz features.baseceZdZdS)FeatureExceptionN__name__ __module__ __qualname__/Users/tedpeters/Documents/Projects/Claude Projects/Misc Tools/notename-sheets/.venv/lib/python3.11/site-packages/music21/features/base.pyrr%Drrc*eZdZdZdZdZdZdZdS)Featurea7 An object representation of a feature, capable of presentation in a variety of formats, and returned from FeatureExtractor objects. Feature objects are simple. It is FeatureExtractors that store all metadata and processing routines for creating Feature objects. Normally you wouldn't create one of these yourself. >>> myFeature = features.Feature() >>> myFeature.dimensions = 3 >>> myFeature.name = 'Random arguments' >>> myFeature.isSequential = True This is a continuous Feature, so we will set discrete to false. >>> myFeature.discrete = False The .vector is the most important part of the feature, and it starts out as None. >>> myFeature.vector is None True Calling .prepareVector() gives it a list of Zeros of the length of dimensions. >>> myFeature.prepareVectors() >>> myFeature.vector [0, 0, 0] Now we can set the vector parts: >>> myFeature.vector[0] = 4 >>> myFeature.vector[1] = 2 >>> myFeature.vector[2] = 1 It's okay just to assign a new list to .vector itself. There is a "normalize()" method which normalizes the values of a histogram to sum to 1. >>> myFeature.normalize() >>> myFeature.vector [0.571..., 0.285..., 0.142...] And that's it! FeatureExtractors are much more interesting. cZd|_d|_d|_d|_d|_d|_dS)N) dimensionsvectorname description isSequentialdiscreteselfs r__init__zFeature.__init__Xs4     rcdg|jzS)zA Prepare a vector of appropriate size and return r)r r&s r _getVectorszFeature._getVectorsdssT_$$rc8||_dS)z< Prepare the vector stored in this feature. N)r*r!r&s rprepareVectorszFeature.prepareVectorsjs&&(( rct|j} d|z }n#t$rtdwxYw|}t |jD] \}}||z||<||_dS)zM Normalizes the vector so that the sum of its elements is 1. ?zcannot normalize zero vectorN)sumr!ZeroDivisionErrorrr* enumerate)r'sscalartempivs r normalizezFeature.normalizeps    C1WFF  C C C"#ABB B C!!dk** ! !DAq&jDGG s6N)rrr__doc__r(r*r,r7rrrrr)s[,,\   %%% )))     rrcLeZdZdZ d ddZdZdZd dZd Zd Z d d Z d Z dS)FeatureExtractora A model of process that extracts a feature from a Music21 Stream. The main public interface is the extract() method. The extractor can be passed a Stream or a reference to a DataInstance. All Streams are internally converted to a DataInstance if necessary. Usage of a DataInstance offers significant performance advantages, as common forms of the Stream are cached for easy processing. NreturnNonec rd|_d|_||d|_t |dsd|_t |dsd|_t |dsd|_t |dsd|_t |dsd|_ t |ds d|_ dSdS) Nr"r#r$r r%Tr7F) rdatasetDatafeaturehasattrr"r#r$r r%r7)r' dataOrStreamkeywordss rr(zFeatureExtractor.__init__s '+  \""" tV$$ DIt]++ $#D t^,, % $D t\** #"DOtZ(( ! DMt[)) #"DNNN # #rc|\t|dr)r'rBs rr?zFeatureExtractor.setDatasj  # i00 )"<?? ) + (55 # (  $ #rcg}|jdkr/||jddnJt |jD]5}||jddd|6|S)a Return a list of string in a form that is appropriate for data storage. >>> fe = features.jSymbolic.AmountOfArpeggiationFeature() >>> fe.getAttributeLabels() ['Amount_of_Arpeggiation'] >>> fe = features.jSymbolic.FifthsPitchHistogramFeature() >>> fe.getAttributeLabels() ['Fifths_Pitch_Histogram_0', 'Fifths_Pitch_Histogram_1', 'Fifths_Pitch_Histogram_2', 'Fifths_Pitch_Histogram_3', 'Fifths_Pitch_Histogram_4', 'Fifths_Pitch_Histogram_5', 'Fifths_Pitch_Histogram_6', 'Fifths_Pitch_Histogram_7', 'Fifths_Pitch_Histogram_8', 'Fifths_Pitch_Histogram_9', 'Fifths_Pitch_Histogram_10', 'Fifths_Pitch_Histogram_11'] r _)r appendr"replacerange)r'postr5s rgetAttributeLabelsz#FeatureExtractor.getAttributeLabelss  ?a   KK ))#s33 4 4 4 44?++ B B ty00c::@@Q@@AAAA rc||j}|j|_|j|_|j|_|j|_|j|_|S)z` Fill the attributes of a Feature with the descriptors in the FeatureExtractor. )r@r"r#r$r r%)r'r@s rfillFeatureAttributesz&FeatureExtractor.fillFeatureAttributessH ?lGy ".#0!_=rct|_||jdS)a^ Prepare a new Feature object for data acquisition. >>> s = stream.Stream() >>> fe = features.jSymbolic.InitialTimeSignatureFeature(s) >>> fe.prepareFeature() >>> fe.feature.name 'Initial Time Signature' >>> fe.feature.dimensions 2 >>> fe.feature.vector [0, 0] N)rr@rRr,r&s rprepareFeaturezFeatureExtractor.prepareFeatures;yy  ""$$$ ##%%%%%rcdS)zF Do processing necessary, storing result in _feature. Nrr&s rprocesszFeatureExtractor.processs rc|||_|||jr|j|jS)z< Extract the feature and return the result. )rrTrVr7r@)r'sources rextractzFeatureExtractor.extractsV   DK   > % L " " $ $ $|rctt}||||S)aR Return a properly configured plain feature as a placeholder >>> fe = features.jSymbolic.InitialTimeSignatureFeature() >>> fe.name 'Initial Time Signature' >>> blankF = fe.getBlankFeature() >>> blankF.vector [0, 0] >>> blankF.name 'Initial Time Signature' )rrRr,)r'fs rgetBlankFeaturez FeatureExtractor.getBlankFeatures8 II ""1%%% rN)r;r<) rrrr8r(r?rPrRrTrVrYr\rrrr:r:s######.)))&0    &&&$       rr:ceZdZdZd:d;dZdd Zd?d@dZdAdZ dZ dZ dZ dZ dZdZdZdZdZdZdZdZidd d!d"d#d$d%e d&d'd(ed)e d*e d+ed,e d-ed.e d/ed0ed1ed2d3d4d5d6ed7ed8Zd9S)B StreamFormsa A dictionary-like wrapper of a Stream, providing numerous representations, generated on-demand, and cached. A single StreamForms object can be created for an entire Score, as well as one for each Part and/or Voice. A DataSet object manages one or more StreamForms objects, and exposes them to FeatureExtractors for usage. The streamObj is stored as self.stream and if "prepared" then the prepared form is stored as .prepared A dictionary `.forms` stores various intermediary representations of the stream which is the main power of this routine, making it simple to add additional feature extractors at low additional time cost. T streamObj stream.Streamc||_|j/|r ||j|_n|j|_nd|_i|_dSr])r_prepareStreampreparedforms)r'r` prepareStreams rr(zStreamForms.__init__)sQ ; " , $ 3 3DK @ @ $   DM02 rr; KeysView[str]c4|jSr])rekeysr&s rrizStreamForms.keys6sz   rr c2|d}|S)z Common routines done on Streams prior to processing. Returns a new Stream Currently: runs stripTies. F)inPlace) stripTies)r'r`s rrczStreamForms._prepareStream:s '''66 rkeystrc||jvr |j|S|d}|j}tt |D]}d|d|dz}||jvr,|dkr*d|d|}|j|}||}||jvr|j|||}ni|dr?|t dd}|| }ntd|d|||j|<|S) zQ Get a form of this Stream, using a cached version if available. .NrrzgetElementsByClass(zno such attribute: z in ) resplitrdrNlenjoin keysToMethods startswithgetElementsByClassrAttributeError) r'rm splitKeysrdr5subKey previousKeylastKey classToGets r __getitem__zStreamForms.__getitem__Dsb $*  :c? "IIcNN =s9~~&& * *AXXiQ/00F##1uu!hhy!}55 :k2lG$,,,64-g6tXFF##$9:: O")#.C*D*DR*G"H #66zBBIIKK$%M7%M%M%M%MNNN!)DJv  rmidi list[int]cdgdz}t|jtjrt |jj}n|jg}|D]}|dddd}t|D]\}}|t|dz kri|dz}||} t|j |} t| j |} |t| | z xxdz cc<t#t$rYwxYw|S)NrT skipRests skipChordsskipGapsnoNoner) rFrdrScorelistpartsfindConsecutiveNotesr1rsgetattrpitchabsrx) r' algorithmhistorprOr5niNextnNextnValue nextValues r_getIntervalHistogramz!StreamForms._getIntervalHistogrames>c  dmV\ 2 2 $,--EE]OE  A))D593715*77D "$  1s4yy1}$$EE KE$QWi88F ' Y ? ?Ic&9"455666!;6666)%  s C(( C54C5rdc8ddlm}||S)Nr) instrument)music21rpartitionByInstrument)r'rdrs rformPartitionByInstrumentz%StreamForms.formPartitionByInstruments(&&&&&&//999rc4td|DS)Ncg|] }|j Sr) forteClassTnI.0cs r z5StreamForms.formSetClassHistogram..s:::A:::rrr'rds rformSetClassHistogramz!StreamForms.formSetClassHistograms:::::;;;rc4td|DS)Ncg|] }|j Sr)orderedPitchClassesStringrs rrz:StreamForms.formPitchClassSetHistogram..sFFF3FFFrrrs rformPitchClassSetHistogramz&StreamForms.formPitchClassSetHistogramsFFXFFFGGGrci}gd}|D]8}|D]3}||vrd||<t||r||xxdz cc<49|S)N) isTriad isSeventh isMajorTriad isMinorTriadisIncompleteMajorTriadisIncompleteMinorTriadisDiminishedTriadisAugmentedTriadisDominantSeventhisDiminishedSeventhisHalfDiminishedSeventhrr)r)r'rdrrirthisKeys rformTypesHistogramzStreamForms.formTypesHistograms+++  ( (A ( (%''%&E'N&71g&&((('NNNa'NNN  (  rcJt|tjritj}|jD]M}|tjD]+}||||,Nn|tj}|Sr]) rFrrrGrrwMeasureinsertgetOffsetBySite)r'rdrOrms rformGetElementsByClassMeasurez)StreamForms.formGetElementsByClassMeasures h - - ?=??D^ 9 9--fn==99AKK 1 1! 4 4a88889 9 ..v~>>D rcht|tjr|ddS|S)NTF)addPartIdAsGroupremoveRedundantPitches)rFrrchordifyrs r formChordifyzStreamForms.formChordifysD h - - $$!%e%EE E Orc4td|DS)Nc6g|]}t|jSr)float quarterLength)rrs rrz:StreamForms.formQuarterLengthHistogram..s"AAA1ao..AAArrrs rformQuarterLengthHistogramz&StreamForms.formQuarterLengthHistogramsAAAAABBBrc4td|DS)Ncg|] }|j Srrrrs rrz6StreamForms.formMidiPitchHistogram..s0001000rr)r'pitchess rformMidiPitchHistogramz"StreamForms.formMidiPitchHistograms00000111rcdtd|D}dgdz}|D] }||||<|S)Ncg|] }|j Sr) pitchClassrs rrz7StreamForms.formPitchClassHistogram..s444qal444rr r)r'rccrks rformPitchClassHistogramz#StreamForms.formPitchClassHistogramsL 44G444 5 5b  A!uE!HH rc,|dS)Nr)r)r'unuseds rformMidiIntervalHistogramz%StreamForms.formMidiIntervalHistograms))&111rcg}|r|j}n|g}|D]}|dddd}t|D]\}}|t |dz kr|dz}||} |jr%|jdj} n |j j} | jr%| jdj} n | j j} | | | z |S)NTFrrrq) hasPartLikeStreamsrrr1rsisChordsortDiatonicAscendingrrrrL) r'rdcListrrrOr5rrrpspsNexts rformContourListzStreamForms.formContourLists)  & & ( ( NEEJE . .A))D5:3715*77D"$ . .1D A &&EE KEy*4466>rBGW\}2!&!!>!Fr!J!O!&!1LL"--- .  rcg}|j}|D]7}t|dtjr||8|S)Nelement) secondsMaprFrNotRestrL)r'rdrOrbundles rformSecondsMapzStreamForms.formSecondsMapsO(   $ $F&+T\:: $ F### rcd|D}d|D}dgdz}|D].}|dks|dkrt|}||xxdz cc</|S)Ncg|] }|d S)durationSecondsrrds rrz1StreamForms.formBeatHistogram..s@@@q*+@@@rc2g|]}td|z S)gN@)roundrs rrz1StreamForms.formBeatHistogram..s"888q5??888rr(r)int)r'r secondsListbpmList histogramthisBPMhistogramIndexs rformBeatHistogramzStreamForms.formBeatHistograms@@Z@@@ 88K888C#I  + +G||w}} \\N n % % % * % % % %rflatc*|Sr])flattenrrs rzStreamForms.s!))++rrc|jSr])rrs rrzStreamForms.sQYrnotesc|jSr])rrs rrzStreamForms.s17rgetElementsByClass(Measure)metronomeMarkBoundariesc*|Sr])rrs rrzStreamForms.sQ5N5N5P5PrrrsetClassHistogrampitchClassHistogramtypesHistogramquarterLengthHistogrampitchClassSetHistogrammidiPitchHistogrammidiIntervalHistogram contourList analyzedKeyc.|dS)Nrm)method)analyze)rr[s rrzStreamForms.s%)@)@rtonalCertaintyc*|Sr])r )rfoundKeys rrzStreamForms.s83J3J3L3Lrc|jSr])metadatars rrzStreamForms.sajrc*tj|Sr])rassembleLyricsrs rrzStreamForms.sT-@-C-Cr)rrassembledLyrics beatHistogramNT)r`ra)r;rg)r`r r;r )rmrnr;rar)r;r)rdra)rrrr8r(rircr~rrrrrrrrrrrrrrrurrrr_r_sj$ 2 2 2 2 2!!!!B@::::<<<HHH$      CCC222222!!!F   --.. ** &'D  "#P#P  L  !: 2 6 , !"< !"< 4 !:  @@!" LL#$10$CC*+MMMrr_c@eZdZdZd dZdZd dZdZdZdZ d Z dS) rHz A data instance for analysis. This object prepares a Stream (by stripping ties, etc.) and stores multiple commonly-used stream representations once, providing rapid processing. Nct|tjr||_d|_nd|_||_|||_n|jx}-t |dr|jx}|j |j|_n|j't |jdr|jj|_n\|jNt |jdrt|jj|_n!t|j|_nd|_d|_ d|_ d|_ d|_ g|_g|_g|_|j|dSdS)Nr sourcePathr)rFrrG streamPath_idrArtitlerrn classLabel _classValue partsCountre formsByVoice formsByPart)featureExtractorClassesForParallelRunningsetupPostStreamParse)r' streamOrPathidr2mds rr(zDataInstance.__init__ sW lFM 2 2 +&DK"DOODK*DO >DHHKq,!Z((-#20(&xDHH [ $l)K)K ${-DHH _ (t 55 0t9::t//DH 9;6 ; "  % % ' ' ' ' ' # "rct|j|_g|_t |jdrUt |jj|_|jjD])}|jt|*nd|_|jtj D])}|jt|*dS)z Set up the StreamForms objects and other things that need to be done after a Stream is passed in but before feature extracting is run. Run automatically at instantiation if a Stream is passed in. rrN) r_rrer"rArsrr rLVoice)r'rr6s rr$z!DataInstance.setupPostStreamParseNs!--  4; ( ( !$+"344DO[& 8 8 '' A7777 8 DOV\* 4 4A   # #KNN 3 3 3 3 4 4rc"||_||_dS)ae Set the class label, as well as the class value if known. The class label is the attribute name used to define the class of this data instance. >>> #_DOCS_SHOW s = corpus.parse('bwv66.6') >>> s = stream.Stream() #_DOCS_HIDE >>> di = features.DataInstance(s) >>> di.setClassLabel('Composer', 'Bach') N)rr)r'r classValues r setClassLabelzDataInstance.setClassLabelis%%rc|jt|jr |jdSt|jr&|j||j|_|jS)Nr)rcallablerr&s r getClassValuezDataInstance.getClassValuevsb   #x0@'A'A #dkFY2 D$ % % =$+*A#// <>> di = features.DataInstance('bach/bwv66.6') >>> len(di['flat']) 197 >>> len(di['flat.pitches']) 163 >>> len(di['flat.notes']) 163 >>> len(di['getElementsByClass(Measure)']) 40 >>> len(di['flat.getElementsByClass(TimeSignature)']) 4 )r)voices)r=r"r!re)r'rms rr~zDataInstance.__getitem__sL  )  # # J  $ $z#rNNr]) rrrr8r(r$r,r/r2r=r~rrrrHrHs,(,(,(,(\4446 & & & &    7 7 7$$$<rrHceZdZdS)DataSetExceptionNrrrrrBrBrrrBceZdZdZddZdZdZ ddZdd Zdd Z dd Z dd Z d Z dZ dZddZdZdZdZddZddZdS)DataSetad A set of features, as well as a collection of data to operate on. Comprises multiple DataInstance objects, a FeatureSet, and an OutputFormat. >>> ds = features.DataSet(classLabel='Composer') >>> f = [features.jSymbolic.PitchClassDistributionFeature, ... features.jSymbolic.ChangesOfMeterFeature, ... features.jSymbolic.InitialTimeSignatureFeature] >>> ds.addFeatureExtractors(f) >>> ds.addData('bwv66.6', classValue='Bach') >>> ds.addData('bach/bwv324.xml', classValue='Bach') >>> ds.process() >>> ds.getFeaturesAsList()[0] ['bwv66.6', 0.196..., 0.0736..., 0.006..., 0.098..., 0.0368..., 0.177..., 0.0, 0.085..., 0.134..., 0.018..., 0.171..., 0.0, 0, 4, 4, 'Bach'] >>> ds.getFeaturesAsList()[1] ['bach/bwv324.xml', 0.25, 0.0288..., 0.125, 0.0, 0.144..., 0.125, 0.0, 0.163..., 0.0, 0.134..., 0.0288..., 0.0, 0, 4, 4, 'Bach'] >>> ds = ds.getString() By default, all exceptions are caught and printed if debug mode is on. Set ds.failFast = True to not catch them. Set ds.quiet = False to print them regardless of debug mode. Nrcg|_g|_g|_||_g|_d|_d|_d|_||dS)NFT) dataInstances_featureExtractors_instantiatedFeatureExtractors _classLabelfeaturesfailFastquiet runParalleladdFeatureExtractors)r'rfeatureExtractorss rr(zDataSet.__init__s\#%.0+%    !!"344444rc|jSr])rIr&s r getClassLabelzDataSet.getClassLabels rctj|s|g}|D]>}|j||j|?dS)zh Add one or more FeatureExtractor objects, either as a list or as an individual object. N)r isIterablerGrLrH)r'valuessubs rrNzDataSet.addFeatureExtractorssn  (( XF > >C  # * *3 / / /  / 6 6ssuu = = = = > >rTcg}|r|d|jD]}||z }|j0|r.||jdd|S)a9 Return a list of all attribute labels. Optionally add a class label field and/or an id field. >>> f = [features.jSymbolic.PitchClassDistributionFeature, ... features.jSymbolic.ChangesOfMeterFeature] >>> ds = features.DataSet(classLabel='Composer', featureExtractors=f) >>> ds.getAttributeLabels(includeId=False) ['Pitch_Class_Distribution_0', 'Pitch_Class_Distribution_1', ... ... 'Pitch_Class_Distribution_11', 'Changes_of_Meter', 'Composer'] IdentifierNrJrK)rLrHrPrIrMr'includeClassLabel includeIdrOfes rrPzDataSet.getAttributeLabels s$  & KK % % %5 , ,B B))++ +DD   ',= ' KK(00c:: ; ; ; rcg}|r|d|jD]}||jg|jzz }|j|r|d|S)a Return column labels for discrete status. >>> f = [features.jSymbolic.PitchClassDistributionFeature, ... features.jSymbolic.ChangesOfMeterFeature] >>> ds = features.DataSet(classLabel='Composer', featureExtractors=f) >>> ds.getDiscreteLabels() [None, False, False, False, False, False, False, False, False, False, False, False, False, True, True] NT)rLrHr%r rIrXs rgetDiscreteLabelszDataSet.getDiscreteLabels(sr   KK   5 2 2B R[MBM1 1DD   ',= ' KK    rcg}|r|d|jD]}|dg|jzz }|j|d|S)a Return column labels for the presence of a class definition >>> f = [features.jSymbolic.PitchClassDistributionFeature, ... features.jSymbolic.ChangesOfMeterFeature] >>> ds = features.DataSet(classLabel='Composer', featureExtractors=f) >>> ds.getClassPositionLabels() [None, False, False, False, False, False, False, False, False, False, False, False, False, False, True] NFT)rLrHr rI)r'rZrOr[s rgetClassPositionLabelszDataSet.getClassPositionLabels>sj   KK   5 , ,B UGbm+ +DD   ' KK    rc2t|s/t|t|krtd|>t|s/t|t|krtdt|rK tj|n"#tj$rtdwxYw|gt|z}t|rL tj|n"#tj$rtdwxYw|gt|z}n|dgt|z}t t|D]1}||}||}||}||||2dS)a add multiple data points at the same time. Requires an iterable (including MetadataBundle) for dataList holding types that can be passed to addData, and an equally sized list of dataValues and an equally sized list of ids (or None) classValues can also be a pickleable function that will be called on each instance after parsing, as can ids. zJIf classValues is not a function, it must have the same length as dataListNzJIf ids is not a function or None, it must have the same length as dataListzVclassValues if a function must be pickleable. Lambda and some other functions are not.zNids if a function must be pickleable. Lambda and some other functions are not.)r.rsrBpickledumps PicklingErrorrNaddData)r'dataList classValuesidsr5rcvthisIds raddMultipleDatazDataSet.addMultipleDataTs%% ^MMS%5%555"\^^ ^ O   MMSXX--"\^^ ^ K  8 U [))))' U U U&(TUUU U'-#h--7K C== ) U S!!!!' U U U&(TUUU U%#h--'CC [&3x==(Cs8}}%% ( (A AQBVF LLB ' ' ' '  ( (sB$$C)C>>Dc|jtdd}t|tr|}|j}||j}n|}t||}||j||j|dS)z Add a Stream, DataInstance, MetadataEntry, or path (Posix or str) to a corpus or local file to this data set. The class value passed here is assumed to be the same as the classLabel assigned at startup. NzCcannot add data unless a class label for this DataSet has been set.)r&) rIrBrFrHrrr,rFrL)r'dataOrStreamOrPathr+r&r2dis rrdzDataSet.addDatas   #"UWW W  (, 7 7 9#B AyM#A0R888B ):666 !!"%%%%%rc||jr"tr|S|S)z Process all Data with all FeatureExtractors. Processed data is stored internally as numerous Feature objects. )rMr_processParallel_processNonParallelr&s rrVzDataSet.processs@   . 1 1 .((** *++-- -rc2jD]}j|_j }t jfdjDt |dd}t|\}}}}t j|}|D]@}jdurt |&t |A|_ tjD]G\} }t|jr || |_t|jr || |_HdS)z5 Run a set of processes in parallel. c"g|] }|jf Sr)rK)rrmr's rrz,DataSet._processParallel..s (Z(Z(Z"dm)<(Z(Z(ZrrT)updateFunctionupdateMultiplyunpackIterableN)rFrGr#rLrrM_dataSetParallelSubprocesszip flattenList environLocal printDebugwarnrJr1r.rr) r'rm shouldUpdate outputData featureDataerrorsrfrgr1r5s ` rrozDataSet._processParallelsH$ S SB;?;RB 8 8:~ '(Z(Z(Z(ZtGY(Z(Z(Z(B7C787; *** 14Z0@- V[##F++ % %AzT!!''****!!!$$$$# t122  EAr'' 0!,Q Q   rcg|_|jD]}g}|jD]}|| |}n#t $ry}d|t |g}|jdurt |nt ||j dur|| }Yd}~nd}~wwxYw| ||j |dS)z7 The traditional way: run non-parallel failed feature extractor:TN)rJrFrHr?rY ExceptionrnrLryrzr{rKr\rL)r'r>rowr[ fReturnedr1fLists rrpzDataSet._processNonParallels*  & & &DC9 & & 4    5 " II 5 5 58"c!ffEEzT))$//6666$))%000},, " 2 2 4 4IIIIII 5 9%%%% M  % % % %+ & &sA C A/CC cg}t|jD]\}}g}|j|}|r'|||D])} |r || jz }|| j*|r'|||||s |s|dS|S)zt Get processed data as a list of lists, merging any sub-lists in multidimensional features. r)r1rJrFrLr2r!r/) r'rYrZconcatenateListsrOr5rr6rmr[s rgetFeaturesAsListzDataSet.getFeaturesAsLists  ..  FAsA#A&B %$$$ ' '#'MAAHHQX&&&&  -))++,,, KKNNNN   7NKrcxg}|jD]/}|}||vr||0|S)z7 Return a list of unique class values. )rFr/rL)r'rOrmr6s rgetUniqueClassValueszDataSet.getUniqueClassValuessJ$  B  ""A}} A rc$ddlm}|dvr||}n\|dvr||}n/|dvr||}ndS|S)Nr outputFormats)taborange taborangeN)dataSet)csvcomma)arff attribute)music21.featuresrlowerOutputTabOrange OutputCSV OutputARFF)r' featureFormatr outputFormats r_getOutputFormatzDataSet._getOutputFormat s222222    $H H H(888FFLL  " "&6 6 6(2242@@LL  " "&; ; ;(33D3AALL4rcd}d|vr\||dd.||dd}|S)a Get an output format from a file path if possible, otherwise return None. >>> ds = features.DataSet() >>> ds._getOutputFormatFromFilePath('test.tab') >>> ds._getOutputFormatFromFilePath('test.csv') >>> ds._getOutputFormatFromFilePath('junk') is None True Nrprq)rrr)r'fpofs r_getOutputFormatFromFilePathz$DataSet._getOutputFormatFromFilePaths\ "99$$RXXc]]2%677C**288C==+<== rrcT||}|S)zS Get a string representation of the data set in a specific format. )r getString)r' outputFmtrs rrzDataSet.getString)s) ,,Y77 %%'''rc||||}n&|td||}|td|d||||S)z/ Set the output format object. NzSpecify a format or a file pathz1no output format could be defined from file path z or format )rrY)rr;rrBwrite)r'rformatrYrs rrz DataSet.write2s >bn<?? ?0088L  "$>&($>$>5;$>$>?? ?!!R;L!MMMr)Nr)TTrr]r@)TTT)r)NNT)rrrr8r(rQrNrPr]r_rjrdrVrorprrrrrrrrrrDrDsY:5555&    > > >48%)8,,-(-(-(-(b&&&&4...   >&&&:4      &((((NNNNNNrrDcg}g}|jD]}|}|| |}ng#t$rZ}|dt |zdzt |z|r||}Yd}~nd}~wwxYw||||||fS)Nrz: ) r#r?rYrrLrnr\r/r2) dataInstancerKrrfeClassr[rr1s rrvrvDs C F I WYY <    - II - - - MM5B?$FQO P P P **,,IIIIII  - 9  2244l6H6H6J6J JJsA B' AB""B'c4ddlm}m}td}t |jt |jz}|||||| ddd}|S)a^ returns a list containing ALL currently implemented feature extractors streamInput can be a Stream, DataInstance, or path to a corpus or local file to this data set. >>> s = converter.parse('tinynotation: 4/4 c4 d e2') >>> f = features.allFeaturesAsList(s) >>> f[2:5] [[2], [2], [1.0]] >>> len(f) > 85 True r jSymbolicnativerrF)rYrZr) rrrrDrrOrNrdrVr) streamInputrrdsr[allDatas rallFeaturesAsListr\s32222222 B   B Y ())D1I,J,JJAAJJ{JJLLL""U-249#;;G Nrrcddlm}ddlm}tj|s|g}g}|D]C}|dvr ||jz }#|dvr ||jz }Dtj|s|g}g}|D]i}|}|dd|dd| |jg}|s|S|D]P} | j |vs|dd kr| | Q|S) aT Given one or more :class:`~music21.features.FeatureExtractor` ids, return the appropriate subclass. An optional `library` argument can be added to define which module is used. Current options are jSymbolic and native. >>> features.extractorsById('p20') [] >>> [x.id for x in features.extractorsById('p20')] ['P20'] >>> [x.id for x in features.extractorsById(['p19', 'p20'])] ['P19', 'P20'] Normalizes case: >>> [x.id for x in features.extractorsById(['r31', 'r32', 'r33', 'r34', 'r35', 'p1', 'p2'])] ['R31', 'R32', 'R33', 'R34', 'R35', 'P1', 'P2'] Get all feature extractors from all libraries: >>> y = [x.id for x in features.extractorsById('all')] >>> y[0:3], y[-3:-1] (['M1', 'M2', 'M3'], ['CS12', 'MC1']) r)r)r) jsymbolicall)rr-rrJr) rrrrrSrrOstriprMrLr&) idOrListlibraryrrrOlibflatIds featureIdrOr[s rextractorsByIdrys4+*****''''''  W % %):: 99;;. . . !< <   YY[[- - - !9 9   X & &:G"" OO%%++-- #r"""#r"""y!!!! D   5;;==G # #wqz'7'7'9'9U'B'B KKOOO Krc<t||}|r|dSdS)a Get the first feature matched by extractorsById(). >>> s = stream.Stream() >>> s.append(note.Note('A4')) >>> fe = features.extractorById('p20')(s) # call class >>> fe.extract().vector [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] )rrrN)r)rrebis r extractorByIdrs, (G < < >> s = stream.Stream() >>> s.append(note.Note('A4')) >>> features.vectorById(s, 'p20') [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] N)rrYr!)r`vectorIdrr[s r vectorByIdrs8 *x ) )) 4 4B zt ::<< rcddlm}m}||dkr*d}|jD] }|j|kr|dfcS|dz }!||dkr,d}|jD] }|j|kr|dfcS|dz }!dSdS)a Returns the list index of the given feature extractor and the feature extractor category (jsymbolic or native). If feature extractor string is not in either jsymbolic or native feature extractors, returns None optionally include the extractorType ('jsymbolic' or 'native') if known and searching will be made more efficient >>> features.getIndex('Range') (61, 'jsymbolic') >>> features.getIndex('Ends With Landini Melodic Contour') (18, 'native') >>> features.getIndex('aBrandNewFeature!') is None True >>> features.getIndex('Fifths Pitch Histogram', 'jsymbolic') (70, 'jsymbolic') >>> features.getIndex('Tonal Certainty', 'native') (1, 'native') rrNrrr)rrrrOr") featureString extractorTyperr indexCountr@s rgetIndexrs(32222222  < <  2  Gwyy~.."K0000 !OJJ  9 9 /  Gwyy~.."H---- !OJJt!: 9rcJeZdZdZdZdZdZdZdZdZ dZ d Z d Z d S) Testc:ddlm}d|_tjd}||}|t|dd|t|dd|t|dd |d }|t|d |d }||d ddddddddddddd ||ddddddddddddd ||dddd dddddd |t|d!d|t|d!ddd"|t|d!d#dd$|t|d!dd%d&|t|d!d#d%d&||d!dd'gd(||d'gd)dS)*NrrJcorelli/opus3no1/1graveri$z flat.notesrz'chordify.flat.getElementsByClass(Chord)z9chordify.flat.getElementsByClass(Chord).setClassHistogramr) z3-11z2-4z1-1z2-3z3-9z2-2z3-7z2-5z3-4z3-6z3-10z3-8z3-2z6chordify.flat.getElementsByClass(Chord).typesHistogram") rrrrrrrrrrrz!flat.notes.quarterLengthHistogramt')g?r.g?g@g@g@g?g?rGrBrzpitches.pitchClassHistogram) r rr rrrrrr) /rrrr*r!r&r)rrJmaxDiffr r8rH assertEqualrs)r'rJr2rmchordifiedChordsrs rtestStreamFormsAzTest.testStreamFormsAs$$$$$$ L2 3 3  " "1 % % RZ#... R -..444 R ^,,b111GH -..444NO "$R2bYZ!"1Qq!!"1.. / / / TU,-1')PQ34PR78"12b RR S S S ?@""2AA !-- . . . R[))1--- R[^L9::B??? R[^L9::B??? R[^,IJKKRPPP R[^,IJKKRPPP GQ(EFAAA C C C 9:DDD F F F F Frcddlm}tj}dD])}|t j|*||}||dgddS)Nrr) c4rd-4d#4f#4a#4d#5a5rzpitches.midiIntervalHistogram)rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr) rrJrrGrLrNoterHr)r'rJr2rrms rtestStreamFormsBzTest.testStreamFormsB0s$$$$$$ MOOL # #A HHTYq\\ " " " "  " "1 % % ;<DDD E E E E Erc0ddlm}ddlm}t j}dD])}|tj|*| |}| ||dd||ddS)Nr)pformatr)rrrrrrrrzflat.secondsMapa4[{'durationSeconds': 0.5, 'element': , 'endTimeSeconds': 0.5, 'offsetSeconds': 0.0, 'voiceIndex': None}, {'durationSeconds': 0.5, 'element': , 'endTimeSeconds': 1.0, 'offsetSeconds': 0.5, 'voiceIndex': None}, {'durationSeconds': 0.5, 'element': , 'endTimeSeconds': 1.5, 'offsetSeconds': 1.0, 'voiceIndex': None}, {'durationSeconds': 0.5, 'element': , 'endTimeSeconds': 2.0, 'offsetSeconds': 1.5, 'voiceIndex': None}, {'durationSeconds': 0.5, 'element': , 'endTimeSeconds': 2.5, 'offsetSeconds': 2.0, 'voiceIndex': None}, {'durationSeconds': 0.5, 'element': , 'endTimeSeconds': 3.0, 'offsetSeconds': 2.5, 'voiceIndex': None}, {'durationSeconds': 0.5, 'element': , 'endTimeSeconds': 3.5, 'offsetSeconds': 3.0, 'voiceIndex': None}, {'durationSeconds': 0.5, 'element': , 'endTimeSeconds': 4.0, 'offsetSeconds': 3.5, 'voiceIndex': None}]r) pprintrrrJrrGrLrrrHr)r'rrJr2rrms rtestStreamFormsCzTest.testStreamFormsC@s""""""$$$$$$ MOOF # #A HHTYq\\ " " " "  " "1 % % $5!677':N#72l#344O' 6' 6' 6' 6' 6rcddlm}ddlm}|gdd}|d}d|_|||d d |d d | | |}| d}| |d| dd}| |d| d}| d}td} | | d||| fD]} t!j| dS)Nrrrql1ql2ql4rComposerrFbwv66.6Bachr+rCorelliz//) lineBreakzIdentifier,Unique_Note_Quarter_Lengths,Most_Common_Note_Quarter_Length,Range_of_Note_Quarter_Lengths,Composer//bwv66.6,3,1.0,1.5,Bach//corelli/opus3no1/1grave,8,0.5,3.75,Corelli)rrZzUnique_Note_Quarter_Lengths,Most_Common_Note_Quarter_Length,Range_of_Note_Quarter_Lengths,Composer//3,1.0,1.5,Bach//8,0.5,3.75,Corellirrrz.arff)suffixr)rr)rrJrrrrDrMrNrdrVrrrrry getTempFiler5remove) r'rJrrOrrrOfp1fp2fp3rs rtestDataSetOutputzTest.testDataSetOutputrs$$$$$$222222$334I4I4I8TT    4 4  1222 9 000 , CCC  $ $R ( (||d|++   [ \ \ \||de|<<   Y Z Z Z hheh$$hheh$$&&g&66 C'''S/  B IbMMMM  rcddlm}ddlm}gd}||d}|d}||t j}|| d| |d | |d d |_ | |j 5|ddddS#1swxYwYdS) Nrr)base)p10p11p12p13rrr MonteverdirHandelT)rrJrrrDrNrrGrLElementWrapperrdrK assertRaisesrrV)r'rJrrOrr2s rtestFeatureFailzTest.testFeatureFailsg$$$$$$      888$334E4?AA    4 4  1222 MOO $$T**+++ 1 ... 1 ***   x8 9 9   JJLLL                  sC>>DDcddlm}ddlm}ddlm}m}t d}t|jt|jz}tj }tj }tj } tj } | | || |||||||jD]R} |jD]H} | |  | -#|j|jjf$rYEwxYwSdS)Nr)analysisrrrr)rrrJrrrrDrrOrrGrPartrrLrrdrNrFrHr?rYrr%DiscreteAnalysisException) r'rrJrrrr[ bareStream bareScore singlePart singleMeasurer>r[s rtestEmptyStreamCustomErrorsz Test.testEmptyStreamCustomErrorss$$$$$$$$$$$$66666666  # # # , - -V5M0N0N N]__ LNN []] (( -((($$$ : 9 """$  D7   4   JJLLLL 1 )CED    s+EEEcvddlm}gd}||d}tjdddd}tjd d dd }|d }|||D]}||d|D]}||d|| d| d| ddS) Demonstrating writing out data files for feature extraction. Here, features are used from the jSymbolic library. rrr31r32r33r34r35p1p2p3p4p5p6p7p8p9rrrrp14p15p16p19p20p21r monteverdiz.xmlN+bachr) numberOfPartsrrrrrrrrrr) rrJrr searchrDrNrdrVr)r'rJrOworksMonteverdi worksBachrws r%x_testComposerClassificationJSymbolicz*Test.x_testComposerClassificationJSymbolicsl %$$$$$NNN %334E4?AA!- 55<??&>??     2 2  1222!&) &)+ = =GAtX = =D$DD1BDD 1V <<<< =(4'46 D DGAtX D DD$DD1BDD 1F CCCC D  u %%% u %%% v &&&&&rcddlm}|gdd}|d}||t jd}t jd}|d ffD]9\}}|jD],}d |d |jj } | |d | -:|dffD]9\}}|jD],}d |d |jj } | |d| -:| | d| d| d|d}||t jd} t jd} | dffD]9\}}|jD],}d |d |jj } | |d | -:| dffD]9\}}|jD],}d |d |jj } | |d| -:| | d| d| ddS)z Demonstrating writing out data files for feature extraction. Here, features are used from the jSymbolic library. rrr&rrIrrErGrJrLrrMrNrOrQz&/_scratch/chinaMitteleuropaSplit-a.tabz&/_scratch/chinaMitteleuropaSplit-a.csvz'/_scratch/chinaMitteleuropaSplit-a.arffrFrHrKrPz&/_scratch/chinaMitteleuropaSplit-b.tabz&/_scratch/chinaMitteleuropaSplit-b.csvz'/_scratch/chinaMitteleuropaSplit-b.arffN) rrJrrDrNr r8rRrrSrdrVr) r'rJrOrrTrVrXr"rBrYrUrWs r$x_testRegionClassificationJSymbolicBz)Test.x_testRegionClassificationJSymbolicB2s %$$$$$%33 G G GHSUU     2 2  1222,344&>??!&)* = =GAtX = =D$DD1BDD 1V <<<< =(45 D DGAtX D DD$DD1BDD 1F CCCC D  9::: 9::: :;;;    2 2  1222,344&>?? &)* = =GAtX = =D$DD1BDD 1V <<<< =(45 D DGAtX D DD$DD1BDD 1F CCCC D  9::: 9::: :;;;;;rcBddlm}|gdd}|d}|||dd |d d d |_d |_|| t|jd | t|jdd|jdd}| |j dgdS)NrrrrrrrrrrrTrr) rrJrrDrNrdrMrLrVrrsr!)r'rJrOrfe00s rtestParallelRunzTest.testParallelRun s$$$$$$$334I4I4I8TT    4 4  1222 9 000 , CCC  R[))1--- R[^,,a000{1~a  qc*****rN) rrrrrrrrr#rCrZr\r_rrrrrs0F0F0FdEEE 060606d(((T2\$ $ $ L''''''R:<:<:rps#"""""$$$$$$  333333++++++ 222222&{&77      |4   SSSSSSSSnRRRRRRRRl~~~~~~~~Dkkkkkkkk^     |4   vNvNvNvNvNvNvNvNr KKK0:8888v"    $$$$P`+`+`+`+`+8 `+`+`+V###w 0 1  zNNNGTr