Before Upgrading (READ FIRST!)

Vulnerability Intelligence: PostgreSQL required for AME 4.0

Deployments using or planning to use Vulnerability Intelligence must have a PostgreSQL instance provisioned and prepared before upgrading to AME 4.0. PostgreSQL is required for Vulnerability Intelligence from AME 4.0 onward.

Ensure every search head running AME can connect to PostgreSQL, and prepare separate schema management and application users.

Migrate existing Vulnerability Intelligence data to PostgreSQL while running AME 4.0, before upgrading to AME 4.2. AME 4.2 actively removes the legacy KV collections, and any data remaining in them will be permanently lost.

See PostgreSQL Requirement for AME 4.0 for the upgrade checklist.

Splunk Cloud Python version

If you are on Splunk cloud run the following query before upgrading.

| rest /services/server/settings splunk_server=local | table python.version

If you get the output python2 - do not upgrade. Ask Splunk support to update the stack to Python3.

info

Python2 EOL was January 1, 2020, but there are still Splunk Cloud instances with Python 2.7 running.

For more information regarding python in Splunk Cloud see:

• https://www.splunk.com/en_us/blog/platform/removing-python-2-from-new-splunk-cloud-and-splunk-enterprise-releases-starting-september-2021.html
• https://docs.splunk.com/Documentation/Splunk/latest/Python3Migration/AboutMigration

If you are upgrading from an AME Version below 3.0

Rename objects with forbidden characters

Since AME 3.0, there are strict naming requirements for the following Objects.

• Templates
• Rules
• Statuses
• Notifications

Please check that they DO NOT contain any of the following characters: < > : ; { } [ ] , ' " \ . and rename them before upgrading.

The pre-upgrade validation script in the next chapter will also check automatically check for these inconsistencies.

Run the pre-upgrade validation script

AME 3.0 and higher has stricter consistency checks for objects. In the pre 3.0 versions, it was possible to delete objects that were still referenced somewhere. To check for inconsistencies, download the following script: ame-validate-3-0-upgrade.py

Execute the script with python3.

The following arguments are valid. Use either username/password or a token for authentication:

python3 validate_3_0_upgrade.py [--url=url] [--username=username] [--password] [--token]

--url       API URL of the AME instance - including port and protocol
--username  Username to authenticate with
--password  Password to authenticate with
--token     Token to authenticate with

The command should return without errors:

$ python3 validate_3_0_upgrade.py --url https://x.x.x.x:8089 --token ey...c3LA
Fetched data from the AME instance

 No known inconsistencies found in status options. Ready for upgrade.
 No known inconsistencies found in notification schemes. Ready for upgrade.
 No known inconsistencies found in rules. Ready for upgrade.
 No known inconsistencies found in templates. Ready for upgrade.
 ...

note

Python3 is required with the request module. If splunk is installed, you can run the command with e.g.: $SPLUNK_HOME/bin/splunk cmd python3 ./ame-validate-3-0-upgrade.py

Before upgrading AME, there should be no errors. Please get in touch with AME support for assistance if any consistency errors are reported you are not able to fix yourself.

Migrate Legacy Non-Slackapp Notifications to Slack-App

Legacy incoming webhooks are no longer supported in AME 3.0 and higher. It recommended that you migrate to app-based webhooks before migration. See Slack Documentation:

Migrating to app-based webhooks

Migrate Time-to-Live Auto-Resolve Settings

Previously, an event with a TTL had a fixed next status of resolved.

With AME 3.0 and higher, it is now possible to define a target status for events that have Time-to-Live (TTL).

Before updating AME we recommend either add new fields/attributes or replace the old settings:

• Replace all eval search overrides ame.time_to_auto_resolve with ame.ttl and ame.ttl_target

• Replace all savedsearches.conf overrides action.create_alert.param.time_to_auto_resolve with action.create_alert.param.ttl and action.create_alert.param.ttl_target.

Migrate Notification Templates to the new Jinja based format

Previously it was possible to create a custom email template. With Jinja based templating the format has changed. Please update the template to the new format.

Events can only be assigned to members of a tenant

Previously, it was possible to assign an event to all users. With AME 3.0, only users with the tenant's power user role can be selected for assignment.

Users need to be tenant members to create events

Previously, creating events in a tenant where the user running the alert was not a member was previously possible. Please check that all alerts run with a user who has the tenant power role assigned for which he creates events.