Li Feifei's team proposed ReKep to give robots spatial intelligence and integrate GPT-4o

Deep integration of vision and robot learning.

When two robot hands work together smoothly to fold clothes, pour tea, and pack shoes, plus the 1X humanoid robot NEO that has been making headlines recently, you may There is a feeling that we are beginning to enter the age of robots.

In fact, these silky movements are the product of advanced robotic technology + exquisite frame design + multi-modal large models.

We know that useful robots often require complex and exquisite interactions with the environment, and the environment can be expressed as constraints in the spatial and temporal domains.

For example, if you want a robot to pour tea, the robot first needs to grasp the handle of the teapot and keep it upright without spilling the tea, and then move it smoothly until the mouth of the pot is aligned with the mouth of the cup. Then tilt the teapot at an angle. Here, constraints include not only intermediate goals (such as aligning the mouth of the pot with the mouth of the cup), but also transition states (such as keeping the teapot upright); together they determine the spatial, temporal and other combination requirements of the robot's actions relative to the environment.

However, the real world is complex and how to construct these constraints is a very challenging problem.

Recently, Li Feifei’s team has made a breakthrough in this research direction and proposed ReKep/Relational Keypoint Constraints. Simply put, this method represents the task as a sequence of relationship key points. Moreover, this framework can also be well integrated with large multi-modal models such as GPT-4o. Judging from the demonstration video, this method performs quite well. The team has also released relevant code. This article is written by Wenlong Huang.

• Paper title: ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation

• Paper Address: https://rekep-robot.github.io/rekep.pdf

• Project website: https://rekep-robot.github.io

• Code address: https://github.com/huangwl18/ReKep

Li Feifei said that this work demonstrates a deeper integration of vision and robot learning! Although the paper does not mention World Labs, an AI company focusing on spatial intelligence founded by Li Feifei in early 5 this year, ReKep clearly has great potential in spatial intelligence.

Method

Relationship Key Point Constraint (ReKep)

First, let’s look at a ReKep instance. It is assumed here that a set of K key points has been specified. Specifically, each keypoint k_i ∈ ℝ^3 is a 3D point on the scene surface with Cartesian coordinates.

A ReKep instance is a function like this: ?: ℝ^{K×3}→ℝ; it can map a set of key points (denoted as ?) into an unbounded cost, When ?(?) ≤ 0, the constraint is satisfied. As for the specific implementation, the team implemented the function ? as a stateless Python function that contains NumPy operations on key points, which may be non-linear and non-convex. Essentially, a ReKep instance encodes a desired spatial relationship between keypoints.

However, an operation task usually involves multiple spatial relationships and may have multiple time-related phases, each of which requires different spatial relationships. To this end, the team's approach is to decompose a task into N stages and use ReKep to specify two types of constraints for each stage i ∈ {1, ..., N}:

• A set of sub-goal constraints

• A set of path constraints

where encodes a key point relationship to be achieved at the end of stage i, and encodes a key point relationship to be satisfied for each state within stage i. Take the tea pouring task in Figure 2 as an example, which consists of three stages: grabbing, aligning, and pouring tea.

The Phase 1 subgoal constraint is to reach the end effector toward the teapot handle. The stage 2 sub-goal constraint is to keep the mouth of the teapot above the mouth of the cup. Additionally, the stage 2 path constraint is to keep the teapot upright to avoid spilling the tea. The final stage 3 subgoal constraint is to reach the specified tea pouring angle.

Use ReKep to define the operation task as a constrained optimization problem

Use ReKep to convert the robot operation task into a constrained optimization problem involving sub-goals and paths. The end-effector pose is denoted here as ? ∈ SE (3). In order to perform the operation task, the goal here is to obtain the overall discrete-time trajectory?_{1:T}:

That is, for each stage i, the optimization The goal of the problem is: given a given set of ReKep constraints and auxiliary costs, find an end-effector pose as the next subgoal (and its associated time), and the pose sequence that achieves this subgoal. This formula can be considered as direct shooting in trajectory optimization.

Decomposition and algorithm instantiation

In order to solve the above formula 1 in real time, the team chose to decompose the overall problem and only focus on the next sub-goal and achieve the The corresponding path of the sub-goal is optimized. Algorithm 1 gives the pseudocode of this process.

The solution formula for the sub-goal problem is:

The solution formula for the path problem is:

Backtracking

The real environment is complex and changeable. Sometimes during the task, the sub-goal constraints of the previous stage may no longer hold (for example, when the tea cup is poured) taken away), this time you need to re-plan. The team's approach is to check the path for problems. If problems are found, iteratively go back to the previous stage.

Forward model of key points

In order to solve equations 2 and 3, the team used a forward model h, which can be used during the optimization process Estimate Δ? based on Δ?. Specifically, given a change in the end-effector pose Δ?, the change in key point position is calculated by applying the same relative rigidity transformation ?′[grasped] = T_{Δ?}・?[grasped], while assuming other key The point remains still.

Key point proposal and ReKep generation

In order to allow the system to freely perform various tasks in real situations, the team also used large models! Specifically, they designed a pipeline process for keypoint proposal and ReKep generation using a large visual model and a visual-linguistic model.

Key point proposal

Given an RGB image, DINOv2 is first used to extract patch-level features F_patch. Bilinear interpolation is then performed to upsample the features to the original image size, F_interp. To ensure that the proposals cover all relevant objects in the scene, they used Segment Anything (SAM) to extract all masks M = {m_1, m_2, ... , m_n} in the scene.

For each mask j, cluster the mask features F_interp[m_j] using k-means (k = 5) and cosine similarity measures. The centroids of the clusters are used as candidate keypoints, which are then projected to world coordinates ℝ^3 using a calibrated RGB-D camera. Other candidates within 8cm of the candidate keypoint will be filtered out. Overall, the team found that this process can identify a large number of fine-grained and semantically meaningful object regions.

ReKep generation

After obtaining the candidate key points, they are then superimposed on the original RGB image and labeled with numbers. Combined with the language instructions of the specific task, GPT-4o is then queried to generate the number of required stages and the sub-goal constraints and path constraints corresponding to each stage i.

Experiment

The team verified the constraint design through experiments and tried to answer the following three questions:

1. The How well does the framework's automated build and composition operations behave?

2. システムは新しいオブジェクトや操作戦略にどの程度うまく一般化できますか?

3. さまざまなコンポーネントがシステム障害にどのような影響を与える可能性がありますか?

ReKep を使用して 2 つのロボット アームを操作

彼らは、多段階 (m)、現場/実践シナリオ (w)、両手操作 (b) とリアクション (r) の動作。これらのタスクには、お茶を注ぐ (m、w、r)、本を並べる (w)、缶をリサイクルする (w)、箱にテープを貼る (w、r)、洗濯物をたたむ (b)、靴を梱包する (b)、共同で折りたたむ (b、 r）。

結果を表 1 に示し、成功率データを報告します。

全体として、新しく提案されたシステムは、タスク固有のデータや環境モデルが提供されない場合でも、正しい制約を構築し、非構造化環境で実行できます。特に、ReKep は各タスクの中核となるパズルを効果的に処理します。

実際の実行プロセスのアニメーションをいくつか示します:

操作戦略の一般化

チームは、洗濯物をたたむタスクに基づいて、新しい戦略の一般化パフォーマンスを調査しました。端的に言えば、システムがさまざまな種類の衣類を折りたためるかどうかを確認することです。これには、幾何学と常識的な推論が必要です。ここでは

GPT-4o が使用されており、プロンプトにはコンテキスト例のない一般的な手順のみが含まれています。 「戦略の成功」は、生成された ReKep が実行可能であることを意味し、「実行の成功」は、衣類の種類ごとに、特定の実行可能な戦略のシステム成功率を測定します。

結果は興味深いものです。システムは衣服ごとに異なる戦略を使用しており、衣服をたたむ方法の一部は人間が一般的に使用するものと同じであることがわかります。

システムエラーの分析

フレームワークの設計はモジュール式であるため、簡単にシステムエラーの解析に便利です。チームは、表 1 の実験で発生した障害ケースを手動で検査し、これに基づいて、パイプライン プロセスにおける時間的な依存関係を考慮して、モジュールがエラーを引き起こす可能性を計算しました。結果を図 5 に示します。

さまざまなモジュールの中で、キー ポイント トラッカーが最も多くのエラーを生成することがわかります。これは、頻繁かつ断続的なオクルージョンによってシステムが正確に追跡することが困難になるためです。

The above is the detailed content of Li Feifei's team proposed ReKep to give robots spatial intelligence and integrate GPT-4o. For more information, please follow other related articles on the PHP Chinese website!