LLMLingua: 整合LlamaIndex，压缩提示并提供高效的大语言模型推理服务

# Install dependency. !pip install llmlingua llama-index openai tiktoken -q   # Using the OAI import openai openai.api_key = "<insert_openai_key>"</insert_openai_key>

!wget "https://www.dropbox.com/s/f6bmb19xdg0xedm/paul_graham_essay.txt?dl=1" -O paul_graham_essay.txt

from llama_index import (VectorStoreIndex,SimpleDirectoryReader,load_index_from_storage,StorageContext, )  # load documents documents = SimpleDirectoryReader(input_files=["paul_graham_essay.txt"] ).load_data()

index = VectorStoreIndex.from_documents(documents)  retriever = index.as_retriever(similarity_top_k=10)  question = "Where did the author go for art school?"  # Ground-truth Answer answer = "RISD"  contexts = retriever.retrieve(question)  contexts = retriever.retrieve(question)  context_list = [n.get_content() for n in contexts] len(context_list)  #Output  #10

# The response from original prompt from llama_index.llms import OpenAI  llm = OpenAI(model="gpt-3.5-turbo-16k") prompt = "\n\n".join(context_list + [question])  response = llm.complete(prompt) print(str(response))  #Output The author went to the Rhode Island School of Design (RISD) for art school.

from llama_index.query_engine import RetrieverQueryEngine from llama_index.response_synthesizers import CompactAndRefine from llama_index.indices.postprocessor import LongLLMLinguaPostprocessor  node_postprocessor = LongLLMLinguaPostprocessor(instruction_str="Given the context, please answer the final question",target_token=300,rank_method="longllmlingua",additional_compress_kwargs={"condition_compare": True,"condition_in_question": "after","context_budget": "+100","reorder_context": "sort", # enable document reorder,"dynamic_context_compression_ratio": 0.3,}, )

retrieved_nodes = retriever.retrieve(question) synthesizer = CompactAndRefine()  from llama_index.indices.query.schema import QueryBundle   # postprocess (compress), synthesize new_retrieved_nodes = node_postprocessor.postprocess_nodes(retrieved_nodes, query_bundle=QueryBundle(query_str=question) )  original_contexts = "\n\n".join([n.get_content() for n in retrieved_nodes]) compressed_contexts = "\n\n".join([n.get_content() for n in new_retrieved_nodes])  original_tokens = node_postprocessor._llm_lingua.get_token_length(original_contexts) compressed_tokens = node_postprocessor._llm_lingua.get_token_length(compressed_contexts)

print(compressed_contexts) print() print("Original Tokens:", original_tokens) print("Compressed Tokens:", compressed_tokens) print("Comressed Ratio:", f"{original_tokens/(compressed_tokens + 1e-5):.2f}x")

next Rtm's advice hadn' included anything that. I wanted to do something completely different, so I decided I'd paint. I wanted to how good I could get if I focused on it. the day after stopped on YC, I painting. I was rusty and it took a while to get back into shape, but it was at least completely engaging.1]  I wanted to back RISD, was now broke and RISD was very expensive so decided job for a year and return RISD the fall. I got one at Interleaf, which made software for creating documents. You like Microsoft Word? Exactly That was I low end software tends to high. Interleaf still had a few years to live yet. [] the Accademia wasn't, and my money was running out, end year back to thelot the color class I tookD, but otherwise I was basically myself to do that for in993 I dropped I aroundidence bit then my friend Par did me a big A rent-partment building New York. Did I want it Itt more my place, and York be where the artists. wanted [For when you that ofs you big painting of this type hanging in the apartment of a hedge fund manager, you know he paid millions of dollars for it. That's not always why artists have a signature style, but it's usually why buyers pay a lot for such work. [6]  Original Tokens: 10719 Compressed Tokens: 308 Comressed Ratio: 34.80x

response = synthesizer.synthesize(question, new_retrieved_nodes) print(str(response))  #Output #The author went to RISD for art school.