The top row is for userassignable functions. Commonlyused parameters are accessed via a simple matrix in which a sixway switch lines up a row of options for tweaking via three small knobs. When inspecting the content of encoded_seq, you will notice that the first token is indexed with 0, denoting the beginning-of-sequence token (in our case, the embedding token). The original Microkorg's geeky panel text and cryptic characters have been superseded by the XL's moody amber display. # Note that the output has a weird list output that requires to index with 0. Tokenizer = om_pretrained('distilroberta-base')Įncoded_seq = tokenizer.encode("i am sentence")įeature_extraction = pipeline('feature-extraction', model="distilroberta-base", tokenizer="distilroberta-base")įeatures = feature_extraction("i am sentence") Now, while the documentation of the FeatureExtractionPipeline isn't very clear, in your example we can easily compare the outputs, specifically their lengths, with a direct model call: from transformers import pipeline, AutoTokenizer There is a discussion within the community about which method is superior (see also a more detailed answer by stackoverflowuser2010 here), however, if you simply want a "quick" solution, then taking the token is certainly a valid strategy. To explain more on the comment that I have put under stackoverflowuser2010's answer, I will use "barebone" models, but the behavior is the same with the pipeline component.īERT and derived models (including DistilRoberta, which is the model you are using in the pipeline) agenerally indicate the start and end of a sentence with special tokens (mostly denoted as for the first token) that usually are the easiest way of making predictions/generating embeddings over the entire sequence.
0 kommentar(er)
