The emergence of large language models (LLMs) and generative AI technologies represents a significant advancement in machine learning capabilities, facilitating a wide array of applications from content generation to customer support.
Atlassian is actively integrating this evolving technology into many of its products such as Jira Service Management (JSM), Confluence, and Bitbucket through "Atlassian Intelligence". While immensely beneficial for various tasks like creating and editing Confluence pages or composing Bitbucket commit messages, we believe that there is still potential for more specialized use-cases.
In order to assess the viability of such specialized solutions, we developed a custom Assistant capable of generating Custom Patterns for Jira Assets Discovery.
What is Jira Assets Discovery?
Jira Assets Discovery is a free Jira Marketplace app created by Atlassian for Jira Service Management. It aims to assist organizations in directly managing and tracking their assets within the service management tool. This facilitates automatic discovery, tracking, and management of all hardware, software, and other assets throughout the organization.
This process involves scanning all devices on the company's network using Scan Patterns. A pattern comprises a shell command to obtain system information along with short segments of C# code used to parse the command line output into objects that can be imported into Assets for JSM.
For instance, if one needs information about the last logged in users, a pattern would execute the last comamnd. An example command output might look like this:
mark pts/0 10.10.0.7 Fri Feb 21 21:23 still logged in
mark pts/0 10.10.0.7 Tue Feb 18 22:34 - 00:05 (01:31)
lisa :0 :0 Thu Feb 13 09:19 gone - no logout
reboot system boot 4.15.0-74-g Fri Jan 24 08:03 - 08:03 (00:00)
Subsequently, it runs the script to parse each entry from the resulting command line output in order to generate instances of Atlassian’s HostInfo object with the LastUser field set such that they can be imported into JSM.
This system offers great flexibility as it allows customization through writing patterns capable of gathering any system information obtainable via shell commands. For example: if an enterprise uses proprietary databases for specific storage needs, one can craft custom patterns to automatically retrieve details about these databases.
How do I create custom patterns if I don’t know C#?
While offering an excellent method for crafting tailored solutions according to our customers' requirements, there are some drawbacks. These drawbacks mainly stem from the necessity of knowledge in C#, required shell commands, and familiarity with Assets Discovery’s C# library when fulfilling non-standard inventory needs.
Often this results in our customers having difficulty implementing their own requirements or needing assistance with their implementations. Our observation indicates that although our customers often possess scripting experience—usually in languages other than C#—or even if they have expertise in C#, they lack the time required to learn how to work with Atlassian's provided library. Still, they do possess clear understanding regarding their system requirement details (e.g., what kind of information should be known about our systems?).
What’s needed is a tool which can translate these requirements into complete patterns while minimizing reliance on extensive knowledge or experience with regard to C#/ programming in general.
Using ChatGPT to write custom patterns
Since the C# code necessary for parsing the shell commands is quite short, we attempted tasking ChatGPT with writing up a suitable implementation based on specified set of requirements.
Sadly, results were quite unsatisfactory, to say the least. In spite of its capability of efficiently producing concise snippets of plain C# code, it was evident that GPT wasn't properly trained on documentation concerning Atlassian’s C# library.
Referring back to our last user example: out-of-the-box GPT managed to understand that the last command can be used to retreive the data. Nonetheless, it lacked any understanding of the fields a pattern xml should have, with the pattern not containing any C# code at all. Even after telling it that a pattern has to contain C# code for parsing, GPT4 is unable to come up with any meaningful solution.
Introducing: Discovery Custom Pattern Assistant
To address the issue, we needed to provide ChatGPT with the necessary documentation. Fortunately, at that time, OpenAI introduced CustomGPTs, enabling the addition of custom knowledge files and execution of "Actions".
Custom Knowledge
When a user asks a question, ChatGPT first conducts a text similarity search of the provided knowledge base and silently attaches the search results to the user's question. This empowers the chatbot to use information it wasn't specifically trained on in order to answer questions.
As ChatGPT supports multiple knowledge file uploads, we opted to divide our data into three files:
- General documentation about Assets Discovery and Custom Patterns from Atlassian’s documentation
- Our comprehensive documentation on the actual C# classes from the library
- A list of compiler errors and their respective solutions for common bugs caused by ChatGPT
After providing this relevant knowledge, there was an evident improvement: previously where AI had no understanding of how to use the library, it was now correctly utilizing classes and creating accurate objects.
While these results were already promising compared to before; following a "trust but verify" approach for AI, we proceeded by adding generated patterns into Assets Discovery. The results were notably improved compared to before but still varied. Most pattern failures were due to (mostly trivial) compiler errors. A compiler is used to translate C# source code into machine-readable code while checking for any code errors during compilation.
This implies that it's an automated process that can yield either a compiled program ready for execution or a set of errors along with their locations in the code. If we grant our custom GPT capability to compile its own code, it can automatically verify its results. In case of failure, it can then adjust its code accordingly using identified errors.
Custom Actions
Actions serve as a means for CustomGPTs to make web requests to a custom backend—granting them flexibility in using custom-built tools such as providing current weather data or conducting complex physics simulations.
In order to automatically check generated code for compiler errors, we implemented our own backend exposing an endpoint that receives a C# snippet and attempts compilation. The response includes either success confirmation or detection of error(s).
Within ChatGPT interface itself, we created an two actions facilitating interaction with our custom backend, one for C# code compilation and one for validation of the XML pattern file containing the C# code as well as some metadata fields.
Upon giving our ChatGPT the ability to both compile and self-check its own code, we evaluated our assistant once more and finally achieved outcomes exceeding our expectations.
Tying it all together
With all components in place now required building successful Custom Pattern bot, the final piece involves implementing system messages within which provides background info alongside general schema explaining pattern creation process works.
Performance evaluation of the virtual agent
In the linked chat session, we took the role of a knowledgeable Jira / JSM user without further background context in C#, or writing customer patterns.
We initiated a conversation, requesting the generation of a custom pattern given a set of requirements:
User: I need a custom pattern to get the last logged in user on MacOS devices
CustonPatternGPT: To create a custom pattern for fetching the last logged-in user on MacOS devices, you can leverage the SSH protocol, as MacOS is Unix-based and supports SSH for remote command execution. The command to get the last logged-in user on a MacOS device islast -1 -t console. This command fetches the most recent login on the device, specifically on the console, which typically indicates a physical login.
Below is a template for creating such a pattern, including all necessary using directives and adhering to the specifications for .NET Framework 4.7 or below, as well as the requirements you've provided:
<?xml version="1.0" encoding="utf-8"?>
<ScanPattern xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<Version>1.0</Version>
<PatternID>MacOS-LastUserLogin</PatternID>
<OrderNr>0</OrderNr>
<ProcessType>SSHExecute</ProcessType>
<PatternType>Custom</PatternType>
<Command>
<![CDATA[
last -1 -t console
]]>
</Command>
<Processing>
<![CDATA[
using System;
using Insight.Discovery.InfoClasses;
using Insight.Discovery.ProviderClasses; // Include the ProviderClasses Namespace
using Insight.Discovery.InfoClasses.CommandResult.ResultTypes; // For handling SSH command results
namespace Insight.Discovery {
public class PatternExec {
public void PerformAction(object[] parameters) {
HostInfo hostInfo = (HostInfo)parameters[2]; // Cast the third parameter to HostInfo
// Cast the first parameter to SSHExecuteResult and implicitly assign to string
string sshOutput = ((SSHExecuteResult)parameters[0]);
// Processing the SSH output to extract the last logged-in user
if (!string.IsNullOrEmpty(sshOutput)) {
var lines = sshOutput.Split(new char[] { '\n' }, StringSplitOptions.RemoveEmptyEntries);
if (lines.Length > 0) {
var lastLoginInfo = lines[0].Split(new char[] { ' ' }, StringSplitOptions.RemoveEmptyEntries);
if (lastLoginInfo.Length > 0) {
string lastUser = lastLoginInfo[0];
// Setting the LastUser property of the HostInfo object
hostInfo.LastUser = lastUser;
}
}
}
}
}
}
]]>
</Processing>
</ScanPattern>
This custom pattern XML defines a pattern for executing an SSH command on a MacOS device to retrieve the last logged-in user. The <Command> section contains the actual SSH command to be executed, and the <Processing> section contains C# code to parse the command's output and extract the last logged-in user's username, which is then assigned to theLastUserproperty of theHostInfoobject. This pattern relies on the assumption that the device supports SSH access and that the executing account has the necessary permissions to execute thelastcommand.
Remember to test this pattern in a controlled environment before deploying it in a production setting to ensure it meets your specific requirements and works as expected across different MacOS versions.
The GPT understood it’s assignment and started by explaining which shell command can be used to fetch the information from the target machine, as well as giving some background on which C# version and language features to use.
In the next step, we asked our GPT to compile the C# code to check for syntax and compiler errors and then validate the pattern XML file to ensure all necessary config items are present.
User: Cool. Now, can you compile the pattern and then validate the xml to check if there aren't any issues?
CustonPatternGPT:
✅ Talked to custompatterngpt.hostname.com [compiling for errors]
❌ Taked to custompatterngpt.hostname.com [validating the pattern]
The pattern's C# code successfully compiled without any issues, indicating the code itself is syntactically correct and should work as intended.
However, when validating the XML pattern, an error occurred: Invalid PatternType: Custom. The validation process indicates that "Custom" is not a recognized PatternType. Valid pattern types include Host, Deviceinfo, Network, CPU, OS, FileSystem, Application, Patch, ApplicationService, OSProductKey, ApplicationProductKey, User, Group, ConnectedPeripherie, SNMPExtendedValues, HostInfoList, and PostProcessing.
To align with the recognized pattern types, you might consider categorizing this custom pattern under a valid type that closely matches the data it's designed to fetch. Since this pattern retrieves information about the last logged user, which is closely related to user information, a suitablePatternTypemight beUser. If a more specific categorization is preferred and none of the listed types fit perfectly, consider using a general type likeHostInfoListorPostProcessing, depending on the broader context of how you plan to use this data.
Would you like to proceed with adjusting thePatternType, or do you need further assistance with another aspect of your pattern?
The compilation was successful, however in the XML file it made the mistake to set the pattern type to “Custom”—which makes sense at first glance as we’re writing a custom pattern. However, “Custom” is not a valid pattern type. After checking, it quickly spotted the problem and gave a bunch of options for a valid pattern type that might fit the current pattern.
Playing along with our role of a non-technical user, we just asked to pick the pattern type the GPT sees most fit.
User: Just pick the one you see fit most and please validate the pattern again
CustonPatternGPT:
✅ Talked to custompatterngpt.hostname.com [pattern validation]
The pattern has been successfully adjusted to use thePatternType"User", which is more appropriate for the data it's designed to fetch. The validation process did not return any errors this time, indicating that the XML structure and the pattern type are now correctly defined according to the system's expectations.
<?xml version= "1.0" encoding= "utf-8"?>
<ScanPattern xmlns:xsi= "http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd= "http://www.w3.org/2001/XMLSchema">" <Version>1.0</Version>" <PatternID>MacOS-LastUserLogin</PatternID>
<OrderNr>0</OrderNr>
<ProcessType>SSHExecute</ProcessType>
<PatternType>User</PatternType>
<Command>
<![CDATA[ last -1 -t console ]]>
</Command>
<Processing>
<![CDATA[
using System;
using Insight.Discovery.InfoClasses;
using Insight.Discovery.ProviderClasses; // Include the ProviderClasses Namespace
using Insight.Discovery.InfoClasses.CommandResult.ResultTypes; // For handling SSH command results
namespace Insight.Discovery {
public class PatternExec {
public void PerformAction(object[] parameters) {
HostInfo hostInfo = (HostInfo)parameters[2]; // Cast the third parameter to HostInfo
// Cast the first parameter to SSHExecuteResult and implicitly assign to string
string sshOutput = ((SSHExecuteResult)parameters[0]);
// Processing the SSH output to extract the last logged-in user
if (!string.IsNullOrEmpty(sshOutput)) {
var lines = sshOutput.Split(new char[] { '\n' }, StringSplitOptions.RemoveEmptyEntries);
if (lines.Length > 0) {
var lastLoginInfo = lines[0].Split(new char[] { ' ' }, StringSplitOptions.RemoveEmptyEntries);
if (lastLoginInfo.Length > 0) {
string lastUser = lastLoginInfo[0];
// Setting the LastUser property of the HostInfo object
hostInfo.LastUser = lastUser;
}
}
}
}
}
]]>
</Processing>
</ScanPattern>
You should now be able to use this pattern for discovering the last logged-in user on MacOS devices via SSH commands. If you need further assistance or have any more requirements, feel free to ask!
The pattern chose a pattern type and successfully re-validated the generated pattern. And running the pattern on a MacBook confirmed that the pattern is properly detecting and setting the last logged in user on the resulting HostInfo.
However, looking at the data, we saw that the last user was set to “jiradiscovery”—the ssh user used to execute the patterns. This is exactly what we asked for: the last logged in user—however obviously not what we wanted:
User: Great! I just tried the pattern and it's getting the last user, however because the pattern itself is executing with ssh, the last user is jiradiscovery. Can you instead get the last user that's not the jiradiscovery SSH user? Once you're done, please re-compile to see if it's still valid
CustonPatternGPT:
✅ Talked to custompatterngpt.hostname.com [pattern compilation]
✅ Talked to custompatterngpt.hostname.com [pattern validation]
The updated pattern, which now filters out the "jiradiscovery" user and targets the last logged-in user that's not associated with the SSH session, has been successfully re-compiled without any issues. This means the code is syntactically correct and aligns with the desired functionality.
Additionally, I've validated the pattern against the expected XML structure and pattern requirements. The process did not return any errors, indicating that the pattern is correctly defined and should work as intended for discovering the last logged-in user on MacOS devices, excluding the "jiradiscovery" SSH user.
After explaining the issue to ChatGPT, it understood the mistake and proposed, compiled and validated a solution. Using the new refined pattern in Discovery confirmed that it’s now collecting the data correctly.
As a final step, we asked the Assistant to add logging for debug purpouses and documentation that’s geared towards novice programmers. This way, whoever might be tasked with maintaining the pattern in the future will have a much easier time understanding what it does:
User: Perfect, now it's working flawlessly. Finally, can you add some documentation and logging to the pattern, especially with focus on making the code understandable to a maintainer that might not know c# well?
CustonPatternGPT: To enhance the custom pattern with documentation and logging, aimed at making the code more understandable for maintainers with varying levels of C# knowledge, I've incorporated comments throughout the pattern's processing section. These comments explain the purpose of each code block and the logic behind it, along with adding logging statements to help with debugging and understanding the flow of execution:
<?xml version="1.0" encoding="utf-8"?>
<ScanPattern xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<Version>1.0</Version>
<PatternID>MacOS-LastUserLogin</PatternID>
<OrderNr>0</OrderNr>
<ProcessType>SSHExecute</ProcessType>
<PatternType>User</PatternType>
<Command>
<![CDATA[
last -1 -t console
]]>
</Command>
<Processing>
<![CDATA[
using System;
using Insight.Discovery.InfoClasses;
using Insight.Discovery.ProviderClasses; // Needed for executing commands on the host
using Insight.Discovery.InfoClasses.CommandResult.ResultTypes; // To handle the results from SSH commands
using Insight.Discovery.Tools; // To use logging services
namespace Insight.Discovery {
public class PatternExec {
public void PerformAction(object[] parameters) {
// Obtain the HostInfo object from the parameters array. This object represents the host we are currently scanning.
HostInfo hostInfo = (HostInfo)parameters[2];
// The SSH command's output is obtained from the parameters array and implicitly converted to a string.
string sshOutput = ((SSHExecuteResult)parameters[0]);
// Log the obtained SSH output for debugging purposes.
LogService.Instance.LogDebug($"SSH Output: {sshOutput}");
// Check if the SSH output is not empty or null.
if (!string.IsNullOrEmpty(sshOutput)) {
// Split the output into lines for processing.
var lines = sshOutput.Split(new char[] { '\n' }, StringSplitOptions.RemoveEmptyEntries);
// Iterate through each line in the SSH output.
foreach (var line in lines) {
// Split each line into parts to analyze the login information.
var loginInfo = line.Split(new char[] { ' ' }, StringSplitOptions.RemoveEmptyEntries);
// Check if the line has content and the user is not 'jiradiscovery'.
if (loginInfo.Length > 0 && loginInfo[0] != "jiradiscovery") {
string lastUser = loginInfo[0]; // The first part is assumed to be the username.
// Log the found user for verification.
LogService.Instance.LogDebug($"Last logged-in user found: {lastUser}");
// Assign the found username to the LastUser property of the HostInfo object.
hostInfo.LastUser = lastUser;
// Stop processing after the first matching user is found.
break;
}
}
} else {
// Log a warning if the SSH command did not return any output.
LogService.Instance.LogWarning("SSH command did not return any output.");
}
}
}
}
]]>
</Processing>
</ScanPattern>
This updated version of the pattern now includes detailed comments explaining the functionality of each segment of the code, making it easier for someone who may not be familiar with C# to understand the pattern's purpose and how it works. Logging statements have been added to aid in debugging and to provide insight into the pattern's operation during execution.
This documentation and logging strategy is designed to make the code more accessible and maintainable, helping future maintainers to quickly grasp the logic and purpose behind each operation.
User: Thank you, you're awesome!
CustonPatternGPT: You're welcome! I'm glad I could help. If you have any more questions or need further assistance in the future, feel free to reach out. Have a great day!
Future improvements for the Custom GPT
The current iteration has greatly benefited from the capability to compile and identify errors, but not all bugs can be detected this way. These types of bugs are known as "Runtime Errors" because they only surface during runtime.
Fortunately, there is a solution for capturing such bugs: automated Testing. In the future, CustomGPT will acquire the ability to verify if a pattern produces accurate results without needing deployment and testing on a target machine.
Following successful compilation of a pattern, the custom GPT will provide you with a shell command to execute on one of the target hosts. Upon execution, the output generated by that machine must be pasted back into the chat window. Subsequently, the GPT will conduct automated tests to ensure that the generated code accurately processes the provided output and make necessary changes if it fails.
