Upgrade your system

This topic explains upgrade procedures for systems that are already ASMS A32.00 with CentOS 7.

It describes how to use the ASMS distributed system upgrade on single appliances, HA/DR clusters, and distributed systems.

Important: For instructions to migrate to CentOS 7, see Upgrade/migration to A32.00 CentOS 7

Note: Before you start, review the upgrade prerequisites and ensure that your system complies. For more details, see Upgrade prerequisites.

Tip: You can check your system's compliance with prerequisite requirements by running upgrade readiness checks prior to upgrading your system. First download build files to your system. See Download ASMS software packages. Then, in the algosec_conf menu, enter option 17 System health, and enter 3 Check upgrade readiness.

Note: Traffic logs that are sent to ASMS during the upgrade procedure may be discarded.

Perform an distributed ASMS upgrade

Distributed ASMS upgrades are supported for standalone hardware or VM appliances, HA/DR clusters, and distributed systems.

Warning: CTRL + C is not supported during the upgrade process, and upgrades cannot be aborted.

Make sure to reserve time for your system upgrade to complete. For more details, see Downtime requirements for upgrades.

Do the following:

  1. Determine the builds that you need to upgrade, and download the relevant software packages from the AlgoSec portal. For details, see Download ASMS software packages.

  2. Access your appliance as user: root

    Note: If you are working on clusters or distributed nodes, access the primary node on the master / Central Manager appliance.

    The upgrade is performed across all nodes in the entire system, starting with the Central Manager.

  3. Copy the downloaded software packages to the following directory: /root/AlgoSec_Upgrade/

  4. Optional/Recommended: In addition, copy the downloaded software packages to the /root/AlgoSec_Upgrade/ directory of remote agents and HA/DRs for which communication is slow. For details see Pre-provisioning of upgrade files on remote nodes.

  5. If you aren't already connected to the ASMS Administration interface (algosec_conf), connect now. For details, see Connect to and Utilize the Administration Interface.

  6. In the administration interface main menu, enter 8 to select Upgrade software.

    Note: The system checks your prerequisites to verify that your system is ready for the upgrade. If any of the prerequisite checks fail, relevant errors are displayed to notify you. In such cases, we recommend making changes so that your system complies with the prerequisite requirements, and then starting the upgrade process again.

    The system lists the available builds from the files you saved in step 3, and prompts you to select the build you want to install. For example:

    ************************************

    *** Software upgrade is starting ***

    ************************************

    Select an AlgoSec build to install:

    1. algosec-appliance-3200.0.0-529-el6.x86_64.run

    2. fa-3200.0.0-891.x86_64.run

    3. Run All

    Note: The option numbering may differ depending on your system configuration.

  7. Do one of the following:

    Run all installations together (recommended)

    Select the option to Run All.

    Note: The option to Run all does not appear at all if you have more than one build per packaged saved. In this case, to run all installations together, first remove the earlier builds.
    Run each installation separately

    Enter the line number for the build you want to install. When each upgrade is complete, start the process again to run the next installation. If you do this, install the builds in the following order:

    1. Appliance build
    2. AFA build
    3. FireFlow

    The system displays details about the upgrade it is about to perform, and prompts you to approve.

    For example:

    The following AlgoSec packages are going to be upgraded:

    * algosec-appliance-3200.0.0-432-el6.x86_64 TO
    algosec-appliance-3200.0.0-529-el6.x86_64

    * fa-3200.0.0-890.x86_64 TO fa-3200.0.0-891.x86_64

    ********************

    *** Upgrade plan ***

    ********************

    Local node : 10.23.0.41

    Remote Agent nodes: 10.23.0.40

    Runtime Estimation: Up to 80 minutes

    Review the upgrade plan detailed above. Approve plan? (y/n):

  8. Prerequisite checks are run including a check for new reports that may need to be synced.

    Note: If errors are discovered, we recommend you stop now and follow the suggested steps. When done, run the upgrade again by going to the algosec_conf menu, and run option 8 - upgrade software. See Resolve automatic system prerequisites checks issues.

  9. If all checks pass, continue by entering y. The upgrade starts.

    If you are working on a distributed system, the upgrade first starts on the local node and then continues with the distributed nodes. The system displays confirmation details as the downloaded packages are copied to the distribution nodes and installed.

    When the upgrade is complete, any clusters are resumed if relevant, and the following message appears:

    *** Software upgrade finished successfully ***

  10. In case of a kernel upgrade on an appliance build, the system also prompts you to reboot. Reboot your system as prompted.  

    Warning: Not rebooting at this stage leaves you with a legacy kernel, which may present security issues.

  11. After upgrading your system, run a report on 'All Firewalls' to ensure a valid network map.

Back to top

External syslog

Note: Java 11 is mandatory for your log collection functionality on your external syslog-ng server in A32.00. In addition, we recommend that your external syslog is running on CentOS 7.

See Upgrade external syslog server to Java 11 and Option 1: To replace an existing external syslog in ASMS.

Back to top

Troubleshoot your distributed upgrade

Back to top

Resolve automatic system prerequisites checks issues

Text in CLI

Description

How to resolve

(log data about prerequisite checks are found in /var/log/algosec-software-upgrade.log) 

Machine [machine IP] does not meet the minimal hardware requirements.

Checks system machine appliance specs: cores, memory.

Make sure the machine meets the system requirements. See Upgrade/migration prerequisites.

For details, see Checking cores and memory on [machine IP] in the log.

 
There is less than xx MB free disk space in OS partition on node [machine IP].

 

Insufficient disk space. xxxMB found for installation (Less than the required 5000 MB in the OS partition on node[machine IP])

 

Partition (/data) on local node must have at least <required> MB free space. This includes the amount of space needed to sync the monitor data directory, plus an additional 5 GB. You currently only have <avail> MB free space.

Checks disk space on system machine.
See Disk space requirements .

Run auto-remove to free up disk or delete old run files.

To run auto-remove, in AFA Administration, go to the Options tab, Storage sub-tab, and click Clean-up now.

If the issue persists after running Clean-up now, contact AlgoSec support.

 
Insufficient disk speed.

Checks source node disk speed.

We recommend disk write speed of at least 300MB/s. Minimum allowable is 80MB/s.

Contact your IT department to determine and adjust, if necessary, your node disk speed.

Tip:

Use the following command to check disk speed:

dd if=/dev/zero of=/data/test-big-file.bin bs=786432000 count=1 oflag=dsync 2>&1 ; rm -f /data/test-big-file.bin

An example of the output is:

786432000 bytes (786 MB) copied, 0.624098 s, 1.3 GB/s

Tip: If your source machine is an AlgoSec VM, make sure you are following VM best practices. See Best practices for your AlgoSec VMware Deployment . If you make changes, check your disk speed again to see if it has improved.

Tip: If your source machine is an AlgoSec AMI, make sure the instance is from the Amazon EC2 General Purpose M4 family (compatible with CentOS 6).

 
Distribution nodes machine time prerequisite check failed.

Compares Time between system machine and distribution nodes (Remote Agent and LDUs).

The machines can be in different time zones but they have to be at the same time relative to UTC:

  1. Compare time and date between CM and the distribution node by running this command on every node mentioned in the message :

    date +%s

    Acceptable results should be up to 180 difference (3 minutes). If a machine exceeds this limit:

    1. Configure NTP server. Use algosec_conf option 2 on the machine to be updated.

    2. Run this command as root user to force time sync:

      ntpdate -u $(awk '$1 =="server"  {print $2}' /etc/ntp.conf)

    3. Reboot the machine.

    4. To verify, rerun on the updated node:

      date +%s
 
NAS is configured, but directories are not mounted.

 

NAS mount is disabled due to fault detected.

Checks NAS status on Central Manager and LDUs.

Open algosec_conf menu on the node with the NAS issue. Run option option 11 - Configure NAS. Run option 3 - Re-enable NAS mount.

If issue persists on an LDU, in the algosec_conf menu, run option 15 - Distributed Architecture configuration.

If problem persists, contact AlgoSec support.

 
NAS is suspended

Open algosec_conf menu on the node with the NAS issue. Run option option 11 - Configure NAS. Run option 3 - Re-enable NAS mount.

If issue persists on an LDU, in the algosec_conf menu, run option 15 - Distributed Architecture configuration.

If problem persists, contact AlgoSec support.

 
The services listed below are not OK.

Checks status of services.

First, try to restart the services. Run for each service:

algosec_test_service -n <SERVICE NAME> -f

for example, algosec_test_service -n postgresql -f

If services do not restart, contact AlgoSec support.

 
Node: 10.20.8.95
* The path /home/afa/algosec should be non-broken symlink
 Checks essential redirect links. Contact AlgoSec support. 
Validation of upgrade files xxx failed. The files may be corrupted. Download the files again.
 Checks for corrupted run files. Download run files again. 
Distribution Architecture is not configured properly.
 Checks for improperly configured distribution nodes. In the algosec_conf menu, run option 15 - Distributed Architecture configuration. 
[product] version earlier than [version #] found on this machine.
 Checks for product versions earlier than two versions before the version you want to upgrade to. Remove the product run file /root/Algosec_Upgrade/<product run file> or upgrade the product to a version not earlier than two versions before the version you want to upgrade to. 
PostgreSQL is not synced between Cluster machine ([machine IP]) and the Primary machine ([machine IP]).
 Checks PostgreSQL sync status between cluster machine and Primary. in the algosec_conf menu, go to option 13 - HA/DR Setup. Select 1 - View cluster status details. 
Inconsistencies found between the devices list and database records.
Checks for database inconsistencies.

To fix the inconsistency, see procedure in the knowledge base article: www.algosec.com/r/a32.00/42845777.

 
FireFlow configuration discrepancy. The FireFlow_configured parameter is set to 'no' but a FireFlow installation .run file was found.
Checks for FireFlow installation inconsistencies.

  • To enable FireFlow: Set the value of the FireFlow_configured parameter to 'yes' in /home/afa/.fa/config. Make sure that FireFlow and AFA run file build numbers match.

  • To disable FireFlow: Delete the FireFlow installation .run file in the /root/AlgoSec_Upgrade directory,

Back to top

Resolve upgrade failures

  • If your distributed upgrade fails for any reason, the system displays an error, as well as the location of specific log files.

    • The central upgrade log file is located at: /var/log/algosec-software-upgrade.log

    • The system also prompts you with options to start the upgrade again.

  • If you have a distributed system and only some nodes failed, you can select the nodes you want to reinstall, or rerun the entire upgrade from scratch. Select the option that works best for you and run through the CLI process as prompted and described above.

  • For HA/DR Suspend/Resume Cluster errors: Go to /var/log/algosec_hadr/ms-hadr.log and check the log for errors.

  • For run file errors: Check the log displayed in the error message for details on why the upgrade failed.

Contact AlgoSec Support for additional assistance, if needed, and send copies of all supporting log information.