php editor Youzi will introduce in this article how to use Python to convert dot-separated values (such as "key1.subkey1.subkey2") into a structure in the Go language. This transformation process is useful for extracting and processing data from configuration files or API responses. We will use Python's recursive functions and Go language structures to implement this conversion, and give detailed code examples and explanations. After studying this article, readers will be able to easily process and convert point-separated values, improving the efficiency and flexibility of data processing.
This is a specific requirement for an application that can change the configuration (specifically the wso2 identity server, since I'm using go to write the kubernetes operator for it). But it's really not relevant here. I want to create a solution that allows easy management of large number of configuration maps to generate go structures. These configurations are mapped in .csv
Link to .csv - my_configs.csv
I want,Write a python script that automatically generates go structuresso that any changes to the application configuration can be updated by simply executing the python script to create the corresponding go structure. I'm referring to the configuration of the application itself. For example, toml key names in csv can be changed/new values can be added.
So far I have successfully created a python script thatalmost achieves my goal. The script is,
import pandas as pd def convert_to_dict(data): result = {} for row in data: current_dict = result for item in row[:-1]: if item is not none: if item not in current_dict: current_dict[item] = {} current_dict = current_dict[item] return result def extract_json_key(yaml_key): if isinstance(yaml_key, str) and '.' in yaml_key: return yaml_key.split('.')[-1] else: return yaml_key def add_fields_to_struct(struct_string,go_var,go_type,json_key,toml_key): struct_string += str(go_var) + " " + str(go_type) + ' `json:"' + str(json_key) + ',omitempty" toml:"' +str(toml_key) + '"` ' + "\n" return struct_string def generate_go_struct(struct_name, struct_data): struct_name="configurations" if struct_name == "" else struct_name struct_string = "type " + struct_name + " struct {\n" yaml_key=df['yaml_key'].str.split('.').str[-1] # base case: generate fields for the current struct level for key, value in struct_data.items(): selected_rows = df[yaml_key == key] if len(selected_rows) > 1: go_var = selected_rows['go_var'].values[1] toml_key = selected_rows['toml_key'].values[1] go_type=selected_rows['go_type'].values[1] json_key=selected_rows['json_key'].values[1] else: go_var = selected_rows['go_var'].values[0] toml_key = selected_rows['toml_key'].values[0] go_type=selected_rows['go_type'].values[0] json_key=selected_rows['json_key'].values[0] # add fields to the body of the struct struct_string=add_fields_to_struct(struct_string,go_var,go_type,json_key,toml_key) struct_string += "}\n\n" # recursive case: generate struct definitions for nested structs for key, value in struct_data.items(): selected_rows = df[yaml_key == key] if len(selected_rows) > 1: go_var = selected_rows['go_var'].values[1] else: go_var = selected_rows['go_var'].values[0] if isinstance(value, dict) and any(isinstance(v, dict) for v in value.values()): nested_struct_name = go_var nested_struct_data = value struct_string += generate_go_struct(nested_struct_name, nested_struct_data) return struct_string # read excel csv_file = "~/downloads/my_configs.csv" df = pd.read_csv(csv_file) # remove rows where all columns are nan df = df.dropna(how='all') # create the 'json_key' column using the custom function df['json_key'] = df['yaml_key'].apply(extract_json_key) data=df['yaml_key'].values.tolist() # read the 'yaml_key' column data = pd.dataframe({'column':data}) # convert to dataframe data=data['column'].str.split('.', expand=true) # split by '.' nested_list = data.values.tolist() # convert to nested list data=nested_list result_json = convert_to_dict(data) # convert to dict (json) # the generated co code go_struct = generate_go_struct("", result_json) # write to file file_path = "output.go" with open(file_path, "w") as file: file.write(go_struct)
The problem is (see below part of csv),
authentication.authenticator.basic authentication.authenticator.basic.parameters authentication.authenticator.basic.parameters.showAuthFailureReason authentication.authenticator.basic.parameters.showAuthFailureReasonOnLoginPage authentication.authenticator.totp authentication.authenticator.totp.parameters authentication.authenticator.totp.parameters.showAuthFailureReason authentication.authenticator.totp.parameters.showAuthFailureReasonOnLoginPage authentication.authenticator.totp.parameters.encodingMethod authentication.authenticator.totp.parameters.timeStepSize
Here, since thebasic
andtotp
fieldsparameters
are duplicated, the script confuses itself and generates twototpparameters
structures. The expected result is to havebasicparameters
andtotpparameters
structures. There are many similar duplicate words in theyaml_key
column of the csv.
I know this has to do with the index being hardcoded to 1 ingo_var = selected_rows['go_var'].values[1]
, but it's hard to fix this.
Can someone give me an answer? I think,
Thanks!
I also tried using chatgpt, but since this has to do with nesting and recursion, the answer provided by chatgpt is not very efficient.
renew
I found a problem with the row containing theproperties
,pooloptions
,endpoint
andparameters
fields. This is because they are duplicated in theyaml_key
column.
I was able to resolve this issue. However, I had to use a completely new approach to the problem, which was to use a tree data structure and then iterate over it. This is the main logic behind it -https://www.geeksforgeeks.org/level-sequential tree traversal/
This is the working python code.
import pandas as pd from collections import deque structs=[] class TreeNode: def __init__(self, name): self.name = name self.children = [] self.path="" def add_child(self, child): self.children.append(child) def create_tree(data): root = TreeNode('') for item in data: node = root for name in item.split('.'): existing_child = next((child for child in node.children if child.name == name), None) if existing_child: node = existing_child else: new_child = TreeNode(name) node.add_child(new_child) node = new_child return root def generate_go_struct(struct_data): struct_name = struct_data['struct_name'] fields = struct_data['fields'] go_struct = f"type {struct_name} struct {{\n" for field in fields: field_name = field['name'] field_type = field['type'] field_default_val = str(field['default_val']) json_key=field['json_key'] toml_key=field['toml_key'] tail_part=f"\t{field_name} {field_type} `json:\"{json_key},omitempty\" toml:\"{toml_key}\"`\n\n" if pd.isna(field['default_val']): go_struct += tail_part else: field_default_val = "\t// +kubebuilder:default:=" + field_default_val go_struct += field_default_val + "\n" + tail_part go_struct += "}\n\n" return go_struct def write_go_file(go_structs, file_path): with open(file_path, 'w') as file: for go_struct in go_structs: file.write(go_struct) def create_new_struct(struct_name): struct_name = "Configurations" if struct_name == "" else struct_name struct_dict = { "struct_name": struct_name, "fields": [] } return struct_dict def add_field(struct_dict, field_name, field_type,default_val,json_key, toml_key): field_dict = { "name": field_name, "type": field_type, "default_val": default_val, "json_key":json_key, "toml_key":toml_key } struct_dict["fields"].append(field_dict) return struct_dict def traverse_tree(root): queue = deque([root]) while queue: node = queue.popleft() filtered_df = df[df['yaml_key'] == node.path] go_var = filtered_df['go_var'].values[0] if not filtered_df.empty else None go_type = filtered_df['go_type'].values[0] if not filtered_df.empty else None if node.path=="": go_type="Configurations" # The structs themselves current_struct = create_new_struct(go_type) for child in node.children: if (node.name!=""): child.path=node.path+"."+child.name else: child.path=child.name filtered_df = df[df['yaml_key'] == child.path] go_var = filtered_df['go_var'].values[0] if not filtered_df.empty else None go_type = filtered_df['go_type'].values[0] if not filtered_df.empty else None default_val = filtered_df['default_val'].values[0] if not filtered_df.empty else None # Struct fields json_key = filtered_df['yaml_key'].values[0].split('.')[-1] if not filtered_df.empty else None toml_key = filtered_df['toml_key'].values[0].split('.')[-1] if not filtered_df.empty else None current_struct = add_field(current_struct, go_var, go_type,default_val,json_key, toml_key) if (child.children): # Add each child to the queue for processing queue.append(child) go_struct = generate_go_struct(current_struct) # print(go_struct,"\n") structs.append(go_struct) write_go_file(structs, "output.go") csv_file = "~/Downloads/my_configs.csv" df = pd.read_csv(csv_file) sample_data=df['yaml_key'].values.tolist() # Create the tree tree = create_tree(sample_data) # Traverse the tree traverse_tree(tree)
Thank you for your help!
The above is the detailed content of Convert dot-separated values to Go struct using Python. For more information, please follow other related articles on the PHP Chinese website!