Databricks releases AI model SDK for big data analysis platform Spark: one-click generation of SQL and FySpark language chart code

According to news on July 10, Databricks recently released the AI ​​model SDK used by the big data analysis platform Spark. When developers write code, they can use it in English Instructions, the compiler will convert the English instructions into PySpark or SQL language code to improve developer efficiency.

▲ Picture source Databricks website

It is reported that Spark is an open source big data analysis tool with more than 1 billion downloads every year and is used globally 208 countries and territories use .

▲ Picture source Databricks website

Databricks said that although Microsoft’s AI code assistant GitHubCopilot is powerful, the threshold for use is also quite high, and Databricks’ SDK is relatively more powerful. Universal and easier to use.

Databricks claims that they found that large language models understand Spark very well, and the Spark community has contributed a large amount of open, high-quality text for AI learning development over the past 10 years. The Databricks team also relied on a series of learning data provided by the Spark community for training and used LLM to develop an English version of the SDK. Developers can use this SDK to directly issue commands in English and get corresponding results, improving development efficiency.

▲ Picture source Databricks website

IT House found that Databricks also claimed that the English version of the SDK is still in the early development stage and they welcome updates. Multiple users can join to improve this SDK.

The above is the detailed content of Databricks releases AI model SDK for big data analysis platform Spark: one-click generation of SQL and FySpark language chart code. For more information, please follow other related articles on the PHP Chinese website!