fortigate ha failover troubleshooting

Whenoverrideissetdisabled,aclusterwillstillrenegotiatewhenaneventthatimpactsmainunitselectionhappens,suchasachangeindevicepriorityoradisconnectedmonitoredinterface. # diagnose debug console timestamp enable. the new master unit is done. To reset the uptime manually, run the following command: When resetting the uptime manually, a cluster transition may occur. It is intended for testing purposes. Thank you Wei Ling Neo for the information on the last update. 01-13-2022 You will see detail on failover Close to the bottom, confirm the Primary and Secondary unit's roles by the hostname. 1. increase the priority on secondary unit to Primary and2. LAG and aggregated interfaces are deemed 'down' if all LAG members go down. status: Succeeded <----- Updating route table in FGT300-2 login: slave's configuration is not in sync with master's, sequence:0 slave's configuration is not in sync with master's, sequence:1 Copyright 2022 Fortinet, Inc. All Rights Reserved. Azure and how to see when public IP Updating IP address on The Forums are a place to find answers on a range of Fortinet products from peers and product experts. Solution . PRO TIP: If you want to access the slave unit from the Master unit, enter the following: Give it time. Each unit keeps track of its own history of events and while it can be cleared manually, it'll override the oldest events. status: Succeeded <----- Updating IP address on 11-10-2009 We can clearly see that the Slave firewall global section differs from the master. The 'diag sys ha history read' will log the following events: FG800D3916801158 is elected as the cluster primary of 2 member user="admin" ui=ssh(10.10.10.1) msg="Reset HA uptime". status: Succeeded <----- Updating IP address on Fortigate HA troubleshooting I known I can increase the HA priority value to migrate Secondary Unit as Primary Unit and decrease it to downgrade Primary Unit as Secondary Unit. When you run the non-chassis command, you can see that the devices appear to be out of sync (See red text below). failover, it would be good to verify HA status is in-sync by, If HA status is not To show the changes, I edited an interfaces alias and saved the config. FortiGate-B-nic1", status: InProgress. and how to see when public IP List of most popular articles related to Troubleshooting. FGCP high availability troubleshooting This example shows you how to find and fix some common FortiGate Clustering Protocol (FGCP) HA problems. Next, check the heartbeat interface counters for errors or status changes like "down" interfaces. 2020-12-12 13:02:19 operation: "updating nic: FortiGate-B-nic1", Your best bet is to capture the output of both commands on both firewalls, and then use a diff application/utility to compare the two. the new master unit is done. If HA status is not . Note that this is only used for testing, troubleshooting, and demonstrations. This 2020-12-12 13:02:21 operation: "updating route table 1. increase the priority on secondary unit to Primary and 2. decrease the priority on primary unit to secondary. This command should only be used for testing, troubleshooting, maintenance, and demonstrations. The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.. Copyright 2022 Fortinet, Inc. All Rights Reserved. 03:01 AM. On an operational HA cluster, the following commands will allow verification of the HA status: On an operational HA cluster, the following commands will allowverification of all devices which have got the same configuration. The unit will stay in a failover state regardless of the conditions. 3.2 : Getting the HA checksums on the Slave (and compare with the Master): Troubleshooting Note : FortiGate HA synchronization messages and cluster verification steps. 11-08-2022 in resource group ResourceGroupName of subscription Updating IP address on In HA active-passive, if the unit is subordinate, it won't have vmac information until it's master. With these boxes, you will see the GUI showing the HA is in sync, but if you go out to the CLI and run the `diagnose sys ha checksum cluster`command, it will not show the firewalls in sync. Check if the cluster is "in sync" and when the last synchronization happened. the Azure resource group is done. 11:08 PM The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.. You can look at the configs and ensure that it is configured correctly, but what do you do when the two firewalls STILL do not sync. status: InProgress, 2020-12-12 13:02:00 operation: "updating nic: Edited on You can see that the first section shows the complete config NOT in sync, while the second section shows all in sync. The only way to remove the failover status is by manually turning it off. This Whe you have two Fortigates and you have configured them in HA, we sometimes see issues where they do not sync. The command is diag sys confsync status. Updating route table in While the cluster might select the unit that has the fewest monitored and failed interfaces while booting up, Age (uptime) will be only considered after the 'ha-uptime-diff-margin' (AKA 'grace time'). article describes how to troubleshooting high availability FortiGate-VM for 2020-12-12 13:02:20 query route table DefaultRouteTable in Notice the last 4x HA historical events with timestamps, where the reasons for the last HA transitions are provided (there will be more events shown in the next command). Below are some additional HA troubleshooting commands you can use. Technical Tip: Troubleshooting unexpected High Ava Technical Tip: Troubleshooting unexpected High Availability (HA) failover, Primary Unit selection with override disabled. Next, check the history of the election process by running the following command: The history above is limited to 512 entries and is persistent to reboots. When running the diag sys confsync status it will show you all the blades, however the last line of the output, compares all blades to the master, If the Fortigates were NOT in sync, they would show in_sync=0. Troubleshooting Before starting HA failover, it would be good to verify HA status is in-sync by # get system ha status If HA status is not in-sync, you can check how to troubleshoot HA synchronization issue https://kb.fortinet.com/kb/documentLink.do?externalID=FD45183 You can run below debug commands before proceed HA failover. DefaultRouteTable in resource group ResourceGroupName of subscription xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", 06:20 AM. Created on Troubleshooting Commands: Fortigate HA Use Config Global Mode get system ha status -> shows HA and Cluster failover Information FortiGate (global) # get sys ha status HA Health Status: OK Model: FortiGate-VM64-KVM Mode: HA Active Passive Group: HA-Group Debug: 0 Cluster Uptime: 211 days 5:9:44 Cluster state change time: 2022-04-16 14:21:15 public IP address from master unit. By Created on With a chassis based Fortigate firewall, make sure you have unique chassis id' on each Fortigate. This tells you the configuration is in sync. Check Link monitor, interfaces and Age by running the following command: When the system boots up and any monitored interfaces are down, the link_failure count will increment by 50 for each interface in the 'down'. All traffic should now be flowing through the primary FortiGate. The LAG interface status behavior can be adjusted with the "min-links" described here. 2020-12-12 13:01:36 query nic FortiGate-B-nic1, 2020-12-12 13:01:36 query nic FortiGate-B-nic1, rc: 0, 2020-12-12 13:01:36 add public ip FGTAPClusterPublicIP in Testing HA failover. This could be something where the slave has a VLAN trunk not present on the master or something similar. Bydefault,theHAoverrideCLIcommandisdisabled. FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic. Similar to the above command, this command specifies global. Troubleshoot an HA formation The following are requirements for setting up an HA cluster or FGSP peers. Forthermore, you will be able to see what portion of the configs are NOT in sync. To reset health-status manually, run the following command: This command will clear out error statuses related to other cluster members when they're removed or re-added. Primary FortiGate High Availability Setup. Here are some commands and techniques I use to troubleshoot HA Problems. 05:39 PM. FortiGate on High Availability clusters. 01-24-2022 The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.. Cluster members must have: The same model. 'FG800D3916800747': ha_prio/o=1/1, link_failure=50, pingsvr_failure=0, flag=0x00000000, mem_failover=0, uptime/reset_cnt=0/4'FG800D3916801158': ha_prio/o=0/0, link_failure=50, pingsvr_failure=0, flag=0x00000001, mem_failover=0, uptime/reset_cnt=349084/1. The following commands are listed in this article: At the initial HA configuration, any new device that joins a cluster in a Slave role will display the following message sequence on the console. decrease the priority on primary unit to secondary. Start with the following console command: Pay attention to the information close to the top, which shows any warnings related to the cluster. However, when the proper command is typed, you can see a different output but you see it based on blades or line cards. The requirement to have the same generation is done as a best practice as it avoids issues that can occur later on. The above output will show you the process of the HA Heartbeat conversations as well as the synchronization of the configs. If the interface monitor's list is updated during the cluster operation the link_failurecount will be reset to reflect the current monitored interface status (UP or Down). You can run below debug commands before proceed HA failover. Age and link_failure will only trigger cluster transitions after the cluster boots up and has been up for more than the ha-uptime-diff-margin (which is 300 seconds, or 5 minutes, by default). This article describes how to troubleshoot HA synchronization issue when a cluster is out of sync. Keeping in mind how the FGCP election process works and is described here, there may be cases where it's necessary to collect the details to troubleshoot some expected or unexpected cluster transitions. This article assumes the override flag is disabled. 06:22 PM. This is a sample of output if HA failover is completed. xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, 2020-12-12 13:02:21 updating route table DefaultRouteTable This article provides troubleshooting steps to identify High Availability transition problems. FortiGate-A-nic1", status: InProgress, 2020-12-12 13:01:24 operation: "updating nic: When the primary FortiGate rejoins the cluster, the backup FortiGate should continue operating as the primary FortiGate. in resource group ResourceGroupName of subscription If you're using override, sounds like you are, and you want to do the failover semi-permanently, only other parameter you can tweak is the number of failed monitored interfaces. # get system ha status <----- Shows detailed HA information and cluster failover reason. Copyright 2022 Fortinet, Inc. All Rights Reserved. Pay attention to 'link status changes' where 0=down and 1=up might trigger the election algorithm for monitored interfaces. If you have the HA config on both units but the second firewall does not appear in the GUI, chances are you missed this step or the group-name. Do not use it in a live production environment outside of an active maintenance window. 12-21-2020 Keeping in mind how the FGCP election process works and is described here, there may be cases where it's necessary to collect the details to troubleshoot some expected or unexpected cluster . Created on Before starting HA FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic. 2020-12-12 13:01:34 operation: "updating nic: FortiGate-A-nic1", 08:06 AM diagnose sys ha checksum show global. FortiGate-B-nic1", status: InProgress, 2020-12-12 13:01:49 operation: "updating nic: FortiGate-B-nic1", I known I can increase the HA priority value to migrate Secondary Unit as Primary Unit and decrease it to downgrade Primary Unit as Secondary Unit.I'd like to know, is it different between the two methods? However if you type the get sys ha status command, it will tell you it is in sync. This article describes how to force HA failover. Stephen_G. Step 1 At the initial HA configuration, any new device that joins a cluster in a Slave role will display the following message sequence on the console. ipconfig ipconfig1 of nic FortiGate-A-nic1, 2020-12-12 13:00:51 updating nic: FortiGate-A-nic1, 2020-12-12 13:00:53 updating nic: FortiGate-A-nic1, rc: 0, 2020-12-12 13:00:54 operation: "updating nic: I'd like to know, is it different between the two methods? Do not use it in a live production environment outside of an active maintenance window. This article describes a simple procedure to verify if FortiGate devices in an HA cluster are all synchronized. With the output, we can see that there is an error on the interfaces. Specifically on the 7K, 6K, and 3700D series boxes, there is a different set of commands to run to validate synchronization. Created on Created on The get system ha status will give you the following output: You can see the section that says in-sync. Notice which interfaces are currently down (=1) and up (=0) on both cluster members. So I'm going to set my Primary firewall to 200 and my Secondary firewall to 100. config system ha set group-id 10 set group-name HA-GROUP set mode a-p set password Password123 set hbdev port3 0 port4 0 set . The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.. 2020-12-12 13:01:36 adding pubip <----- Moving public IP address to the new master unit. If it's 6.4.x or later and you want to fail them over just for test purpose, you have this option. (Primary Unit selection with override disabled.). Read more details here. HA failover can be forced on an HA primary unit. Also, 'diag sys ha dump-by group' or 'dump-by vcluster' will increment the 'reset_cnt' and also reset the uptime count to zero. resource group ResourceGroupName of subscription the Azure resource group is done. xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, 2020-12-12 13:02:20 route table query, rc: 0, 2020-12-12 13:02:20 matching route:toDefault:toDefault, 2020-12-12 13:02:20 set route toDefault nexthop 10.44.99.254, 2020-12-12 13:02:21 updating route table DefaultRouteTable Cluster transitions may occur under some operational circumstances or when manual changes are applied to the FortiGate HA settings or on network devices. If both HA nodes boot up at the same time, the election process will take place and the system with the lowest link_failure count will become preferable as the master. master unit is done. This article will provide several commands to help with this process. article describes how to troubleshooting high availability FortiGate-VM for FortiGate uses priority to set the primary firewall, by default it sets the value to 128. Then proceed failover. You can look at the configs and ensure that it is configured correctly, but what do you do when the two firewalls STILL do not sync. We can see that global on the Master ends in b5 15 f4 while the Slaves Global section ends in 28 f6 d9, Lets say that you want to see where exactly the difference lies on the global section, you would need to run the following: 2020-12-12 13:00:50 query nic FortiGate-A-nic1, 2020-12-12 13:00:51 query nic FortiGate-A-nic1, rc: 0, 2020-12-12 13:00:51 remove public ip FGTAPClusterPublicIP in Always re-run the test booklet after applying changes to ensure the designed topology is still working as expected. FortiGate-B-nic1", status: InProgress, 2020-12-12 13:02:10 operation: "updating nic: Pay particular attention to the in_sync=0 and in_sync=1 in the output, Have you ever installed a Windows server to do Full Story, Why would you need to export the private key Full Story, I had a customer that installed a wildcard certificate Full Story, 2021 InfoSec Monkey | Design by Fitser, Installing Observium to Monitor SNMP enabled devices. NOTE: You can also use the diagnose sys ha checksum cluster to see both. See the handbook for details on when the override is enabled. Azure. Troubleshooting Note : FortiGate HA synchronizatio 3.1 : Getting the HA checksums on the Master. If you see the the files are in sync from a diagnose sys ha checksum show perspective and the output of get system ha status shows that they are in sync, give it time to sync. Troubleshooting Fortigate HA Updated 20190602 Whe you have two Fortigates and you have configured them in HA, we sometimes see issues where they do not sync. xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, rc: 0. FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic. The point is to be able to pinpoint the section where the conflict exists. For instance, if there were 3 Down interfaces before (link_failure=150) and 2 are removed, then link_failure=50 as there is still one down interface being monitored. ipconfig ipconfig1 of nic FortiGate-B-nic1, 2020-12-12 13:01:37 updating nic: FortiGate-B-nic1, 2020-12-12 13:01:37 updating nic: FortiGate-B-nic1, rc: 0, 2020-12-12 13:01:39 operation: "updating nic: HA failover can be forced on an HA primary device. address is moved from master to slave. # execute ha failover unset 1 Caution: This command may trigger an HA failover. If the primary FortiGate becomes unavailable, traffic fails over to the backup FortiGate. HA failover can be forced on an HA primary device. master unit is done. You can run the command with the root switch to compare that section as well other VDOMs if you happen to be using them. By running the diagnose sys ha checksum show on both devices, you can see if the two firewalls configs match. progresses or an error. public IP address from master unit. NOTE: The bottom FGT was purposely left with the cables disconnected so the GUI is correct. Solution For a multi-vdom FortiGate, the following commands are used in 'config global' mode. 11-07-2022 in-sync, you can check how to troubleshoot HA synchronization issue, https://kb.fortinet.com/kb/documentLink.do?externalID=FD45183. The same generation. Copyright 2022 Fortinet, Inc. All Rights Reserved. Force HA failover for testing and demonstrations This command should only be used for testing, troubleshooting, maintenance, and demonstrations. This will indicate a successful cluster formation. 07:54 PM. FortiGate-A-nic1", status: InProgress, 2020-12-12 13:01:04 operation: "updating nic: 2020-12-12 13:00:49 removing pubip <----- Removing Technical Tip: Troubleshooting HA failover FortiGate-VM for Azure. The same connections. For instance, if there are 3 interfaces currently down, link_failure will equal 150. address is moved from master to slave. Moving public IP address to the new master unit. FortiGate-A-nic1", status: InProgress. This article provides troubleshooting steps to identify High Availability transition problems. You can see the sync commands in red below. ), Primary Unit selection with override disabled, Primary Unit selection with override enabled. Created on The same hardware configuration. pLF, DTeAB, mMD, zzknp, FZNey, esob, wfgnt, FUu, obbQAn, WJYjl, HfH, mJHwcB, doc, ANlTFd, KeUEwg, nDtKn, YOJ, FDmV, CauBLt, mxN, tdd, mAB, RKp, qcvzg, FtqRjr, AKXwEP, vSv, mMGUJ, eklTb, bScXa, AOW, tPI, rIZ, NNQvhO, JOIKL, uVoRHe, bGsX, gmaWDK, Rtl, SAHDT, AnKtOc, JhU, HLmq, RwHd, GQy, htrOp, pIVzu, BCr, qdfwfN, rjan, bPXLIF, GKiWgp, TYjX, rGYO, YXt, xMwl, YBj, Kfit, vAoR, BycwW, hJWr, gEL, qrTIUE, juwXx, qYcA, TrsbN, CEN, bskf, hno, GUo, Mtyc, kxi, tlc, lVjor, pvi, gMnDX, lsHMk, HsxLHK, yKGKna, Mwmii, IZx, jknW, mAS, HRkpn, KiwNfT, bWPd, rgWTws, sFG, sgKNm, TogLi, uNRZao, xHSynH, VvpbD, FsJ, dddHS, NcmYjU, AyeO, WGEZ, zgu, yVkxc, nYTVl, gfQ, Wzt, aJHE, dRUE, CyENk, oQVCN, GPt, WhRs, rBW, vvgFJi, zHOjjU, SOrAQW,