與 Jira 和 LLM 的互動專案報告

對於我從事的所有項目，我使用了某種專案管理系統，其中項目範圍被定義為任務清單（票證），並透過更改任務狀態來報告進度。

雖然此類專案管理系統提供各種儀表板和報告，但解釋工程師創建的具有神秘標題的一長串任務並非易事。為了給專案發起人和客戶提供透明度，我必須手動建立專案報告，然後回答相關問題。

如果我們向法學碩士尋求協助怎麼辦？這個想法很簡單：獲取所有專案任務並將它們提供給法學碩士索取報告。然後開始聊天以提出進一步的問題。

收集數據

我們將使用 Jira 進行此實驗，因為它是一個流行的工具，具有易於使用的 REST API。我們將為其創建報告的範例專案非常技術性 - 它是關於創建一個建置腳本，該腳本可以檢測程式碼使用的內容並產生建置系統所需的指令。這樣的專案肯定會有一些標題晦澀難懂的技術任務。

讓我們從獲取任務開始。範例設定中的項目表示為帶有子票證（任務）清單的單父票證（史詩）。對於每個任務，我們將取得完整的歷史記錄，以查看工單狀態如何隨時間變化。使用Python的Jira客戶端實作很簡單。請注意，在 Jira 命名法中，使用術語 issue 而不是 ticket，這反映在程式碼中。

jira = JIRA(server=os.environ["JIRA_SERVER"], basic_auth=(os.environ["JIRA_USER"], os.environ["JIRA_TOKEN"]))

def fetch_issues(epic_key):
    issues = []
    print("Loading epic data...", end="", flush=True)
    issues.append(jira.issue(epic_key))
    print("done")

    print("Loading tasks...", end="", flush=True)
    child_issues = jira.search_issues(f"parent = {epic_key}")
    for issue in child_issues:
        issues.append(jira.issue(issue.key, expand="changelog"))
    print("done")

    return issues

由於取得所有具有歷史記錄的票證需要一段時間，因此可以方便地將這些資料儲存在本地以進行進一步的實驗。在使用實作時，我使用以下函數將任務儲存到檔案並從檔案載入它們：

def save_issues(filename, issues):  
    with open(filename, "x") as file:
        file.write("[")
        file.write(",".join(
            json.dumps(issue.raw) for issue in issues))
        file.write("]")

def load_issues(filename):
    with open(filename, "r") as file:
        data = json.load(file)
        return [Issue(jira._options, jira._session, raw=raw_issue)
            for raw_issue in data]

準備數據

下一步是為LLM準備資料。 JSON 格式的原始 Jira 資料非常冗長，我們不需要所有這些額外的欄位。讓我們提取基本資訊：主題、描述、類型、狀態和創建日期。從歷史記錄中，我們只會提取工單狀態變更及其日期和作者，忽略其他欄位的變更。

所有這些資訊都將以純文字形式儲存。我看過有人使用 JSON 或 XML 作為 LLM 輸入，但我的觀察是 LLM 非常擅長解釋純文字資料。另外，透過這種方法，我無需擔心將文字欄位格式化為 JSON 或 XML 相容。我所做的唯一處理就是從描述中去掉空行，主要原因是為了讓我更容易查看結果。

def strip_empty_lines(s):
    return "".join(line for line in (s or "").splitlines() if line.strip())

def issue_to_str(issue):
    return f"""
{issue.fields.issuetype}: {issue.key}
Summary: {issue.fields.summary}
Description: {strip_empty_lines(issue.fields.description)}
Type: {issue.fields.issuetype}
Status: {issue.fields.status}
Created: {issue.fields.created}
Priority: {issue.fields.priority}
"""

def changelog_to_str(changelog, changeitem):
    return f"""
Author: {changelog.author.displayName}
Date: {changelog.created}
Status change from: {changeitem.fromString} to: {changeitem.toString}
"""


def history_to_str(issue):
    if issue.changelog is None or issue.changelog.total == 0:
        return ""
    history_description = ""
    for changelog in issue.changelog.histories:
        try:
            statuschange = next(filter(lambda i: i.field == "status", changelog.items))
            history_description += changelog_to_str(changelog, statuschange)
        except StopIteration:
            pass
    return history_description

#this function assumes the first issue is an epic followed by tasks.
def describe_issues(issues):
    description = "Project details:"
    description += issue_to_str(issues[0])
    description += "\nProject tasks:"
    for issue in issues[1:]:
        description += "\n" + issue_to_str(issue)
        description += f"History of changes for task {issue.key}:"
        description += history_to_str(issue)
    return description

我用於此實驗的史詩有 30 個任務，這些任務的歷史記錄中有 1 到 15 次狀態變更。我不會引用describe_issues函數的完整輸出，但為了讓您了解它的外觀，這裡有一個簡短的摘錄：

Project details:
Epic: TKT-642
Summary: Create universal build script
Description: 
Type: Epic
Status: In Development
Created: 2024-05-24T10:48:33.050+0200
Priority: P4 - Low

Project tasks:

Task: TKT-805
Summary: add test reporting for e2e tests
Description: 
Type: Task
Status: In Progress
Created: 2024-09-06T09:56:33.919+0200
Priority: P4 - Low
History of changes for task TKT-805:
Author: Jane Doe
Date: 2024-09-06T10:04:15.325+0200
Status change from: To Do to: In Progress

Task: TKT-801
Summary: Sonar detection
Description: * add sonar config file detection *
Type: Task
Status: In Progress
Created: 2024-08-30T13:57:44.364+0200
Priority: P4 - Low
History of changes for task TKT-801:
Author: Jane Doe
Date: 2024-08-30T13:57:58.450+0200
Status change from: To Do to: In Progress

Task: TKT-799
Summary: Add check_tests step
Description: 
Type: Task
Status: Review
Created: 2024-08-29T18:33:52.268+0200
Priority: P4 - Low
History of changes for task TKT-799:
Author: Jane Doe
Date: 2024-08-29T18:40:35.305+0200
Status change from: In Progress to: Review
Author: Jane Doe
Date: 2024-08-29T18:33:57.095+0200
Status change from: To Do to: In Progress

提示

我們將使用的提示由兩個部分組成。首先，它給出了創建報告的說明，詳細說明了我們想要在報告中包含哪些具體資訊。然後我們插入前面段落中準備的門票資訊。法學碩士往往會給出冗長的答复，因此我們特別要求不要添加任何額外的解釋。由於實驗是在終端機中進行的，我們也會要求響應終端友好。

def create_prompt(isses_description):
    return f"""
    Using below information from jira generate executive report that focuses on what was done and what is remaining.
    Provide project progress information. Point out problematic tasks and what may be wrong.
    Use status change history to point out tasks that were worked on for too long or have been reopened many times. 
    Check if you can detect other problems from tasks or their history.
    Provide only text content that can be displayed on terminal, without any additional explanation or markdown.
    {isses_description}
    """

我用於此實驗的法學碩士是 Anthropic 的 Claude 3.5 Sonnet。除了發送提示訊息之外，我們還將指定身份。可以將身份添加為提示開頭的另一行，但由於計劃是跟進問題（創建互動式聊天），因此克勞德的文檔建議在這種情況下將身份設置為單獨的系統參數。

def send_messages(messages):
    client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_KEY"])
    identity = "You are a seasoned IT program manager. You have very good communication skills, your answers are concise and focus on key insights."
    message = client.messages.create(
        system=identity,
        max_tokens=2048,
        messages=messages,
        model="claude-3-5-sonnet-20240620"
    )
    return message.content[0].text

def generate_report(prompt):
    messages=[
        {
            "role": "user",
            "content": prompt 
        }]
    return send_messages(messages)

讓我們把它放在一起並運行：

if __name__=="__main__":
    issues = fetch_issues('TKT-642')
    descrption = describe_issues(issues)
    prompt = create_prompt(description)
    report = generate_report(prompt)
    print(report)

以下是我從克勞德那裡收到的完整報告：

執行報告：通用建置腳本專案 (TKT-642)

專案進度：

• 24 項任務中的 11 項已完成 (45.8%)
• 5 項任務正在進行或正在審核 (20.8%)
• 8 項任務尚未啟動 (33.3%)

主要成就：

1. 為建構器建立了原型 jinja 範本
2. 實現了配置擴展和驗證
3. 新增了單元測試執行步驟
4. 整合聲納掃描支援
5. 創建了用於構建配置和技術腳本的圖像

進行中：

• 新增 e2e 測試的測試報告
• 實現環境和聲納偵測的濾波邏輯
• 開發開發者文件

有問題的領域：

1. Task TKT-734 (Add config extension) went through multiple status changes, indicating potential issues or complexity.
2. Task TKT-697 (Add support for sonar scanning) has been reopened multiple times, suggesting implementation challenges.
3. Task TKT-715 (Create developer documentation) has been in progress for an extended period without completion.

Upcoming Work:

• Integration of the build system into various services (authentication, database-api, shopping-cart, reporting, localization)
• Adding support for monorepos and custom jobs
• Creating user documentation

Recommendations:

1. Review the implementation of config extension and sonar scanning to address recurring issues.
2. Prioritize the completion of developer documentation to support team adoption.
3. Begin planning for the integration of the build system into remaining services.
4. Consider reassessing the priority of tasks that have remained in the "To Do" state for an extended period.

Overall, the project is making steady progress, with nearly half of the tasks completed. However, attention should be given to resolving recurring issues and completing in-progress tasks to maintain momentum.

The result is quite good. It did miscount tickets - in various runs I did it gave me a different number, but never a correct number of 30. Well, LLMs are known for not being able to count r's in the word strawberry. But the summary is correct, recommendations are also reasonable.

Asking questions

Now let's create an interactive session and ask some questions. We'll start with a pre-defined question which is the original prompt we used to generate the report. Then we will read the user's input and attach it to messages. Each response, along with the report is also recorded in the messages chain.

def interactive_session(prompt):
    messages = []
    user_input = prompt
    while user_input != "exit":
        messages.append({
            "role" : "user",
            "content" : user_input
        })
        response = send_messages(messages)
        print(response)
        messages.append({
            "role" : "assistant",
            "content" : response
        })
        user_input = input("QUESTION: ")
    return messages

To run this, just replace generate_report in the main section of the script with interactive_session. Below is a recording of an example session I did. I omitted from the quote the first response which just prints out the report.

QUESTION: how many engineers the team counts?

Based on the information provided in the task histories, I can identify at least three distinct engineers working on this project:

1. John Smith
2. Jane Doe
3. Mike Carpenter

These three names appear frequently in the status change histories for various tasks. However, it's important to note that this may not represent the full team size. There could be other team members who haven't made changes recorded in these specific task histories, or who are involved in the project in other capacities not reflected in this data.

To get an accurate count of the team size, you would need additional information beyond what's provided in these task histories.

QUESTION: is unit test execution task done yet?

Yes, the unit test execution task is completed. Specifically, task TKT-700 with the summary "Add unit tests execution step" is marked as Done. Here's the relevant information:

Task: TKT-700
Summary: Add unit tests execution step
Status: Done

The history of changes for this task shows that it was moved to "Done" status on 2024-08-02, after going through the typical workflow stages (To Do -> In Progress -> Review -> PREPROD VERIFICATION -> READY FOR PRODUCTION -> Done).

QUESTION: does the scope include running some sort of static code analysis tool?

Yes, the project scope includes running static code analysis. Specifically, there's a task for implementing Sonar scanning, which is a popular static code analysis tool. Here are the details:

Task: TKT-697
Summary: Add support for sonar scanning
Status: Done

This task was completed and moved to "Done" status on 2024-09-02. The implementation of Sonar scanning indicates that static code analysis is part of the universal build script project.

Additionally, there's another related task that's currently in progress:

Task: TKT-801
Summary: Sonar detection
Status: In Progress

This task mentions "sonar detection" in its summary, which further confirms that Sonar (a static analysis tool) is being integrated into the build process.

These tasks demonstrate that static code analysis, particularly using Sonar, is included in the project scope and has been largely implemented.

QUESTION: what are the 3 biggest risks of the project?

Based on the provided information, the three biggest risks for this project appear to be:

1. Integration Delays:
   Several tasks for integrating the build system into various services (localization, reporting, shopping cart, authentication, database API) are still in "To Do" status. This poses a risk of delays in widespread adoption of the universal build script across different services. Failure to integrate across all intended services could limit the project's overall impact and value.

2. Documentation Lag:
   The task for creating developer documentation (TKT-715) has only recently moved to "In Progress" after being in "To Do" for an extended period. Additionally, the user documentation task (TKT-714) is still in "To Do" status. This lag in documentation creation risks hindering the adoption, proper use, and maintenance of the new build system. It could lead to misuse, inefficiencies, or resistance to adoption due to lack of clear guidance.

3. Complexity in Implementation:
   Several tasks, particularly those related to Sonar scanning (TKT-697) and config extension (TKT-734), have shown multiple status changes between "In Progress" and "Review". This pattern suggests potential technical challenges or uncertainties in implementation. There's a risk that these complexities could lead to bugs, performance issues, or maintenance difficulties in the future if not properly resolved.

These risks could potentially impact the project's timeline, adoption rate, and overall success if not properly managed and mitigated.

That was surprisingly good. I do like the fact the LLM is using generic knowledge in addition to prompt data, so it was able to answer questions about the "static code analysis tool" even though no ticket data mentions those terms. The answers are a bit too verbose, more experiments with the prompt are required - probably appending instructions to each user's query would help shape better answers.
Issues such as miscounting tickets should be easy to solve, we can calculate base statistics and include them in the prompt itself.

以上是與 Jira 和 LLM 的互動專案報告的詳細內容。更多資訊請關注PHP中文網其他相關文章！