Manage demands and use large models to solve problems: the chart processing artifact SheetCopilot is online

This silky smooth operation process is simply good news for professionals!

Recently, researchers from the Institute of Automation, Chinese Academy of Sciences, Hong Kong Polytechnic and other institutions have created a "Table AI Assistant" "SheetCopilot, this intelligent agent can generate solutions for manipulating tables based on user instructions and execute them on specific software (such as Excel, GoogleSheets, etc.). SheetCopilot can quickly connect to multiple table processing software, and supports multi-table operations, chart drawing and pivot table generation. It is expected to empower table data processing and visualization in multiple fields and take a key step towards realizing a generalist intelligent assistant.

## Website: https://sheetcopilot-demo.github.io/

Paper: https://arxiv.org/abs/2305.19308

Let us first experience how SheetCopilot can significantly improve work efficiency through the following examples.

Suppose you are a newbie and one day your boss asks you to help him analyze sales data. When you get the form and look at it, you'll be dazzled by the thousands of rows of data. You don't know where to start, so you check it as you go.

First, you try to extract the name of each Product, and then use a formula to sum the income of each Product.

After working on it for more than 20 minutes, SUMIF kept reporting "#NAME?" errors, so I gave up.

I continued to search online and found that there is such a convenient tool as a Pivot table, so I started the second challenge.

It took more than twenty minutes to finally get it done. The whole process took nearly an hour, and the efficiency was not satisfactory. Every time your boss gives you a new form processing task, your previous experience is of little use, so you can only view the website and process it from scratch.

Your colleagues use SheetCopilot and can complete all kinds of strange requirements in just a few seconds :).

Seeing that SheetCopilot is so smooth, you gave it a try and it was easy to draw a picture.

With SheetCopilot, you no longer need to slide the mouse across half the table to select data that exceeds the screen. Thousands of rows can be easily selected. Data moves between multiple tables.

Why SheetCopilot was proposed

For a long time, people have longed for someone who can become proficient even without professional experience. Ability to master complex software. Many of us have been in situations where we didn't know how to navigate PhotoShop's cumbersome interface, wanted to analyze data but didn't know the advanced feature of PivotTables, wanted to draw gears but didn't know anything about Solidworks.

With the emergence of large language models (LLMs) with powerful language understanding and generation capabilities, this vision is closer to reality than ever before. If LLM can be guided to master various software, the almost unlimited potential of LLM can be released, thereby allowing human productivity to reach unprecedented heights.

This article points out that the spreadsheet is an ideal basis for conducting this research because it is a common and versatile production tool. However, table manipulation faces diverse challenges, and it is difficult for users to master sufficient table processing and programming skills to cope with ever-changing task requirements.

If there is a generalist AI agent that masters rich software control skills, not only will office efficiency be greatly improved, but enterprise output will also be greatly accelerated. The emergence of SheetCopilot just fits people's vision.

What are the highlights of SheetCopilot

1. Covering the typical needs of table processing

SheetCopilot covers It handles almost all typical tasks of table operations and can excellently complete various chart generation tasks.

Sales data analysis

Experimental chart drawing

##Complex formula calculation

##Applying Conditional Formatting

2. Beyond the VBA-based approach

SheetCopilot is superior to the method of using GPT-3.5 to generate VBA code and execute it. The generated solution is significantly better than the latter (see the figure below), which makes SheetCopilot expected to become a powerful AI auxiliary tool for data processing personnel in the future. .

Not only that, compared to the obscure VBA code, the solution generated by SheetCopilot contains easy-to-understand steps. This eliminates the pain of learning a new programming language and struggling to debug it.

Left picture: lengthy VBA code; right picture: SheetCopilot simple and easy to understand solution plan.

3. Comfortable using experience

SheetCopilot in When the network connection is stable, it only takes about 10 steps of multi-table combination operations to quickly complete the task in tables with thousands of rows and dozens of columns. This not only frees up the user's tired eyes, but also saves the time wasted in finding the website and trying the steps one by one, and also avoids the cost of learning VBA.

Method Principle

This article abstracts the core functions required for table manipulation into a set of virtual APIs (called atomic operations, see the figure below), which are used to generate solutions as LLM A bridge for interaction with application software.

The simplest method is to generate all steps of a task in one query of LLM. However, as the complexity of the task increases, subsequent steps become more dependent on the execution results of previous steps, making it difficult for this kind of open-loop control to obtain correct results. For example, LLM has difficulty determining the scope of an operation if it cannot determine where the data will be visible after filtering.

In order to achieve efficient closed-loop control, SheetCopilot optimizes solutions based on software status feedback and external atomic operation knowledge base, improving success rate and efficiency.

How to evaluate

This article proposes a high-quality evaluation benchmark. The tasks of this benchmark have diverse formulations and involve rich atomic operations, as shown in the following word cloud:

This benchmark adopts the following Metrics about success rate (higher is better):

• Exec@1: Execution success rate of the generated task solution.
• Pass@1: Task pass rate, that is, the proportion of solutions that can match any reference answer after execution.

This benchmark also considers the following efficiency metrics (lower is better):

• A50: Will meet mission requirements The number of steps in the solution is divided by the minimum number of steps in the reference answer, and then the median of all calculations is taken.
• A90: The calculation method is the same as above, but the 90th percentile of all calculation results is taken. This indicator reflects the extreme value distribution of the number of actions.

Experimental results

##Table 1: Comparison on SheetCopilot data set GPT-3.5-Turbo, GPT-4, Claude and how to generate VBA.

Unsurprisingly, GPT-4 has the highest proportion of solutions that meet the task requirements and has the best efficiency, while GPT-3.5-Turbo follows closely behind, with Claude winning the most. times but also close to GPT-3.5-Turbo.

One noteworthy result is that SheetCopilot achieved an excellent success rate compared to the method of translating user instructions into VBA code and executing them on Excel. This means that SheetCopilot brings software intelligent control a big step closer to us, allowing users who cannot program to command computers to complete complex tasks through daily communication.

Let’s take a look at the advantages and disadvantages of each of these three LLMs through the indicators in each subdivision category below.

GPT-3.5 and GPT-4 easily solve Management (table management operations such as sorting and filtering) and Entry & manipulation (data input and manipulation) tasks, both achieved 100% executability rate. In addition, the three LLMs each showed the best efficiency in different task categories. This interesting finding shows that each LLM has its unique advantages and that GPT-4 cannot outperform other models.

Conclusion

SheetCopilot uses LLM to successfully form a closed loop of perception, reasoning and decision-making through text interfaces to achieve efficient spreadsheet control and promote smarter software control. Going to the next level also brings new inspiration to researchers interested in generalist agents.

The above is the detailed content of Manage demands and use large models to solve problems: the chart processing artifact SheetCopilot is online. For more information, please follow other related articles on the PHP Chinese website!