exporter

Overview ¶

Package exporter provides tools for extracting and converting chat session data from JSON files into various formats, such as CSV and JSON datasets.

This package facilitates tasks like data visualization, reporting, and machine learning data preparation.

The exporter package defines types to represent chat sessions, messages, and associated metadata.

It includes functions to:

• Read chat session data from JSON files
• Convert sessions to CSV with different formatting options
• Create separate CSV files for sessions and messages
• Extract sessions to a JSON format for Hugging Face datasets

The package also handles fields in the source JSON that may be represented as either strings or integers by using the custom StringOrInt type.

Additionally, it now supports context-aware operations, allowing for better control over long-running processes and the ability to cancel them if needed.

Code:

func (soi *StringOrInt) UnmarshalJSON(data []byte) error {
	// Try unmarshalling into a string
	var s string
	if err := json.Unmarshal(data, &s); err != nil {
		// If there is an error, try unmarshalling into an int
		var i int64
		if err := json.Unmarshal(data, &i); err != nil {
			return err // Return the error if it is not a string or int
		}
		// Convert int to string and assign it to the custom type
		*soi = StringOrInt(strconv.FormatInt(i, 10))
		return nil
	}
	// If no error, assign the string value to the custom type
	*soi = StringOrInt(s)
	return nil
}

Usage examples:

To read chat sessions from a JSON file and convert them to a CSV format with context support:

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
defer cancel()

store, err := exporter.ReadJSONFromFile("path/to/chat-sessions.json")
if err != nil {
    log.Fatal(err)
}
err = exporter.ConvertSessionsToCSV(ctx, store.ChatNextWebStore.Sessions, exporter.FormatOptionInline, "output.csv")
if err != nil {
    log.Fatal(err)
}

To create separate CSV files for sessions and messages:

err = exporter.CreateSeparateCSVFiles(store.ChatNextWebStore.Sessions, "sessions.csv", "messages.csv")
if err != nil {
    log.Fatal(err)
}

To extract chat sessions to a JSON dataset:

datasetJSON, err := exporter.ExtractToDataset(store.ChatNextWebStore.Sessions)
if err != nil {
    log.Fatal(err)
}
fmt.Println(datasetJSON)

Copyright (c) 2023 H0llyW00dzZ