Custom scripted fields

nbisurgi · July 3, 2025, 9:05pm

Hi,
I created 4 new scripted fields in the pulse-server-state-* index pattern. My question relates to future Pulse updates and whether or not they will stay there.
Currently that index has 96 fields that come by default (plus the 4 new I added); if tomorrow you add a 97th field and ship pulse with it, will it remove my custom ones?
Thanks!
nico

ecarmona · July 4, 2025, 12:31am

Hi @nbisurgi,

Thank you for your message.

It’s generally best to avoid modifying the index definition directly, as Pulse will update or overwrite it during startup to maintain consistency between the index and its internal API. Additionally, the index structure may change in future releases.

If you need to add fields or make changes, feel free to reach out, we’d be happy to consider including them in a future update.

Alternatively, you can create a separate index and use scripts to move data from the Pulse indices to your own.

Kind regards,
Erik

meissa · July 4, 2025, 10:25am

Hi @nbisurgi

I am interested to know how to add scripted fields, May you please share any good documentation ?

Regards,
Muhammad Eissa

nbisurgi · July 10, 2025, 6:30pm

Hi Eric,
Thanks for your response. I took your advice and created a new index pattern for these as well as other inferred metrics.
The scripted fields I created were intended to parse the Context value when Product=‘PAfE’. This is because the client wants to get some insights in terms what workbooks are being used the most, which are generating long running threads, which are generating the most amount of threads, which users are using older versions of PAfE, etc.
For that I used the custom scripted fields to create: PAfEVersion and WorkbookName (since Excel version is handled by IT and they are alll using the same x64 version).
So, something like this:

{"Context":
  "PAfE/2.0.104.7 (0); Excel/16.0.17928 (x64, Consolidated PnL ACTvsFCT.xlsx, Results)"
}

Generated:

{
  "PAfEVersion": "2.0.104.7",
  "WorkbookName":  "Consolidated PnL ACTvsFCT.xlsx"
}

Note that sheet name is not included because it’s deceiving; when checking the behavior it was clear that it displays the ActiveSheet at the moment of the refresh, not the sheet containing the MDX from the ExecuteMDX/Cellset request.

So, we could either add this to the pulse-server-stats-* index or maybe generate another index pattern for PAfE related threads only.

I’m also working on a new reverse lookup index to take the blocking threads as events and will show all the threads that are being locked by it. Though for this one I went directly into Python. You could used saved searches (I think) or get suuuuuper fancy with SQL, but those are not my skillset list

More than happy to chat about any of the above to see if you want to include it in Pulse.

Cheers!

nico

nbisurgi · July 10, 2025, 6:38pm

Hi Muhammad,
To add scripted field you need to:

Go to Explorer → Dashboard Management → Index Patterns
Then choose the index pattern you want to add the scripted field to
Go to the second tab (Scripted Fields) and add a new one.

The language is Painless

if (doc.containsKey('Product') && doc['Product'].size() > 0 && doc['Product'].value == 'PAfE') {
  if (doc.containsKey('Context') && doc['Context'].size() > 0) {
    String context = doc['Context'].value;

    // Convert to lowercase for case-insensitive search
    String lowerContext = context.toLowerCase();

    // Look for excel pattern
    int excelPos = lowerContext.indexOf('excel/');
    if (excelPos > -1) {
      // Find the opening parenthesis after excel/version
      int openParen = context.indexOf('(', excelPos);
      if (openParen > -1) {
        // Find content inside parentheses
        int closeParen = context.indexOf(')', openParen);
        if (closeParen > -1) {
          String parenContent = context.substring(openParen + 1, closeParen);

          // Find first comma (after architecture like x64)
          int firstComma = parenContent.indexOf(',');
          if (firstComma > -1) {
            // Get everything after first comma - this should be workbook name
            String remainder = parenContent.substring(firstComma + 1).trim();

            // If there's another comma, split there (before sheet name)
            int secondComma = remainder.indexOf(',');
            if (secondComma > -1) {
              return remainder.substring(0, secondComma).trim();
            } else {
              // No second comma, return everything after first comma
              return remainder;
            }
          }
        }
      }
    }

    // Return full context for debugging
    return 'Debug: ' + context;
  }
}
return 'Not PAfE or No Context';

This would extract the workbook name for a document where Product=“PAfE”.

Cheers!

nico

ecarmona · July 11, 2025, 2:11am

Hi @nbisurgi and @meissa,

Please avoid creating scripted fields directly on the existing Pulse indices, as we cannot guarantee their persistence over time as Pulse evolves.

Keep in mind that these scripts run concurrently while Pulse is indexing data, which can impact OpenSearch performance. The embedded Pulse cluster uses single-shard indices—meaning only one thread handles reads and writes—which is sufficient for Pulse’s intended use case.

Also, when the Pulse cluster status turns yellow during shard assignment for new indices, Pulse cannot write to OpenSearch. For this reason, please avoid creating follower indices within the same Pulse cluster, as it may cause instability.

An incorrect script or a constantly yellow cluster could jeopardize Pulse’s overall stability.

If you need to implement such scripted fields or follower indices, we recommend deploying a standalone OpenSearch or Elasticsearch cluster. You can then create follower indices on top of the Pulse indices and safely add your custom scripts there.

If you require new fields added to Pulse indices, please open a ticket. We will gladly review it as a potential enhancement.

Thank you for your understanding.

Kind regards,
Erik

meissa · July 11, 2025, 9:00am

Hi @nbisurgi ,

I appreciate your help, I was reading about this and couldn’t understand the steps to achieve it.

Regards,
Muhammad Eissa

meissa · July 11, 2025, 9:02am

Hi @ecarmona ,

I understand your concerns, This was more about knowing than applying. Also it will important to test ideas before requesting the field in a ticket.

Regards,
Muhammad Eissa

Vincent · July 21, 2025, 11:59am

@ecarmona , we should add these two new fields by default.

Thanks @nbisurgi for the suggestion

ecarmona · September 9, 2025, 3:53am

Hi @nbisurgi ,

We’ve included these fields in Pulse’s Opensearch, but it also tracks any product, the version and also the Version, architecture of Excel. Added to the later, you will also see the Workbook and Worksheet.

the name conventions are of those of Pulse. instead of PAfeVersion, it would be ProductVersion.

this how the doucument looks now:

A sample document looks as follows using discover:

for slice:

Now The RAW Document looks like this:

{
“_index”: “pulse-server-state-2025-09”,
“_id”: “ub6LLJkB3uQsQ4DaIPNW”,
“_score”: 0,
“_source”: {
“ServerName”: “CWLINUX”,
“ServiceName”: “tm1srv01”,
“IsSystemThread”: false,
“IsWaiting”: false,
“IsRunning”: true,
“IsIdle”: false,
“IsCommitting”: false,
“IsRollingBack”: false,
“timestamp”: “2025-09-09T13:35:34.954+10:00”,
“Id”: “thread-139888158570048”,
“SessionId”: 48,
“ThreadId”: 139888158570048,
“UserName”: “Admin”,
“DisplayName”: “Admin”,
“State”: “Run”,
“Function”: “VIEW”,
“ObjLockStatus”: “(R)0(IX)0(W)0”,
“Time”: 0,
“SessionTime”: 200080000,
“LastActionTime”: 0,
“ThreadType”: “User”,
“Context”: “Slice/2.1 (0); Excel/16.0.16130 (x64, 07 - BUD Pack - Vx.xlsx, Parameters)”,
“Product”: “Slice”,
“ProductVersion”: “2.1”,
“ExcelVersion”: “16.0.16130”,
“ExcelArchitecture”: “x64”,
“Workbook”: “07 - BUD Pack - Vx.xlsx”,
“Worksheet”: “07 - BUD Pack - Vx.xlsx”,
“ObjectName”: “”,
“ObjectType”: “”,
“RLocks”: 0,
“IXLocks”: 0,
“WLocks”: 0,
“WaitTime”: 0,
“ThreadInfo”: “”,
“DocumentType”: “TM1TopItem”,
“_isPulseThread”: false
}

This is included in the next Release coming very soon!

Kind Regards,

Erik