Click2Gov or Click2Breach?
June 14, 2018 • RBS
Here on the Cyber Risk Analytics research team, we have more than our fair share of “glitch in the matrix moments” – you know, that proverbial black cat walking across your screen that makes you think: “Didn’t I just see this breach?” Usually it’s a case of similar circumstances or simply two names that are a lot alike. Other times, it might be something more.
We have been tracking a handful of breaches taking place across the country that on the surface look to be unique events with somewhat similar descriptions. A local city or town discovers their online utility payment portal has been attacked. The service goes dark while the city investigates – along with their trusty vendor that may or may not run the portal – only to learn that payment card details used to pay utility bills online have been compromised. The city takes responsibility for the event and starts posting notices to impacted persons. All and all, there was nothing especially remarkable about the individual reports – until, that is, the the name Click2Gov started popping up.
What We Know So Far
On May 25, 2018, the City of Oxnard, CA was notified by a bank that their online utility bill payment service appeared to have been breached, leading to a number of fraudulent transactions. Transactions taking place between March 26, 2018 and May 29 (yes, 4 days after the city first learned of the issue – more on that later) were exposed. The city identified Click2Gov as their payment processing application.
On June 6, 2018, the Village of Wellington, FL was notified by Superion that certain vulnerabilities in Click2Gov might have lead to a possible breach of their online utility payment installation. Once again, Wellington officials in conjunction with Superion shut down the system to investigate. While a breach has yet to be confirmed, there was sufficient information for the Village to state that payment card data used for online bill payments between July 2017 and February 2018 is considered to be ‘at risk’.
Two events in a row referencing the same application got our attention and sparked our curiosity. Especially so since the City of Oxnard event began one short month after the Village of Wellington event seemingly ended. Our immediate thoughts went to questions like: “Are there more breaches involving Click2Gov? Could it be the same attackers jumping from one vulnerable installation to the next? Is it possible that the source of the issue is attackers inside Superion, picking off data from various clients?” Definitive answers are not yet apparent, but it is clear that the issue is larger than just two breaches.
Looking back in our database, the City of Ormond Beach, FL experienced a similar incident with their Click2Gov system in October 2017. Like Oxnard, it was a credit card issuer that first traced the issue back to Ormond Beach utility payment system, alerting them of the problem on October 11. This, despite the fact that customers had been reporting fraudulent charges they believed to be linked to the City since September 22nd. Ultimately, cards used for payment between approximately mid-September 2017 and October 4, 2017, when the city opted to shut down their system, may have been compromised.
Shortly after, the City of Port Orange, FL launched their own investigation into their Click2Gov system. Their system was down for 5 days but ultimately, they could find no evidence of a breach. Curiously, their statement included a quote that their Click2Gov system had no “potential flaws that could leave the system exposed to a data breach.” One can only wonder if they are equally confident of no flaws now that Superion has notified at least one customer, the Village of Wellington, of “certain vulnerabilities” in the Click2Gov system.
Our research identified more breaches at several other cities that fit the profile of a Click2Gov issue. The vendor wasn’t named in official statements, but in several instances is clear Click2Gov is source:
- City of Goodyear, AZ – May 7, 2018 the City became aware of an issue with their unnamed online payment system. They worked with the vendor and determined transactions between June 13, 2017 and May 5, 2018 had been exposed. Although the city does not come out and name Click2Gov as the vendor, it’s clear from the payment landing page URL that Click2Gov is the service provider: https://click2gov.goodyearaz.gov/Click2GovCX/index.html
- City of Thousand Oaks, CA – February 28, 2018, the city learned of unauthorized access to their online payment system “Click to Gov”, exposing payment card details for transactions between November 21, 2017 and February 26, 2018.
- City of Fond du Lac, WI – Once again, on December 12, 2017, the city got word from a bank that a breach had been traced back to their water payment portal. Payments made between August 2017 and October 2017 were exposed. Yet again, Click2Gov was not named but is clearly they are the provider of payment services: https://click2gov.fdl.wi.gov/Click2GovCX/index.html
- City of Beaumont, TX – On August 24, 2017, the city announced they had received complaints of unauthorized charges after using the online water bill payment system. Payments made between August 1st and August 24, 2017 may have been “jeopardized”. Beaumont did not indicate a vendor was involved, but it’s clear who their service provider is as well: https://beau-egov.aspgov.com/Click2GovCX/index.html
- City of Oceanside, CA – In near lock step with Beaumont, on August 14, 2017 the city received complaints from customers that credit cards used between June 1, 2017 and August 15, 2017 on the now-defunct “utility bill payment” link had been compromised. The link is no longer available so it is unknown whether it was Click2Gov, but the city’s notification letter does state their forensic examiner found “malicious code had infiltrated this vendor supported online payment system.” Perhaps most telling, the letter goes on to state, “the City is exploring alternative online payment solutions that offer improved security processes and systems.” Clearly a wise decision on their part.
As you can imagine, we suspect there are others.
Unfortunately, we aren’t intimately familiar with how Click2Gov software works exactly. From how the cities are reporting the events, it appears to be a software package that is downloaded and run independently for each city. After all, the cities seem to be taking responsibility for the breach, hiring the forensic teams to investigate and making statements to the effect of updating their software and making changes to servers in response. But further digging seems to reveal that while it is a software package, there may be some vendors that are hosting it on behalf of their clients and the Click2Gov solution may also provide credit card processing capabilities.
What makes this interesting is that, for each incident that has been reported, the breach is presented as some sort of misconfiguration issue or a problem at the city itself, but it seems that it might be something larger.
Despite indications there were issues with the service dating back to August of 2017, it wasn’t until May 30th of this year in the City of Oxnards’ breach notification that we start to see clear evidence the problem lies with Click2Gov – and it’s not encouraging. Oxnard officials posted the following on their Facebook page:
“Upon discovery, the city immediately reported the issue to the Police Department and the city’s vendor, which engaged a third-party forensic firm to determine what happened and what information may have been affected. The city’s vendor alerted the city to a software vulnerability that had the potential to allow an unauthorized individual to gain access to the computer used to process credit card transactions.”
Keep in mind the City of Oxnard first learned of a possible breach on May 25, 2018. They reached out to Superion, seeking help with the issue. Additionally, Superion most likely knew of potential security problems since the City of Oceanside stopped using their service back in the summer of 2017, and certainly since Beaumont, Texas was breached at approximately the same time. Both facts make this next paragraph from Oxnards’ breach notice all the more concerning:
“Security patches were applied by the city’s vendor on a new server to eliminate the vulnerability with the thought that the issue was resolved. On May 29, 2018, the city’s vendor informed the city of additional security controls that were required to secure the system. The city shut down the system immediately so these security controls could be implemented. Even though the vendor’s investigation could not specifically confirm or verify the exact method by which any credit card data could have been compromised, the city decided to notify customers as a precaution.”
Multiple clients are breached over the course of a year and still it takes two tries to get a fix in place? And is the problem really corrected if they cannot confirm or verify the exact method of compromise? Looking back to the City of Fond du Lac’s breach notification, it seems this is not the first time they stumbled over incident response.
“The compromised credit cards each used the City’s online Water Payment Portal at some point approximately between August and October 2017 to pay a City of Fond du Lac water bill.”
“In October 2017, the City’s vendor third party payment engine identified a known vulnerability with the Water Payment Portal. This vulnerability was communicated to the City and patched by the vendor on the same day. The City received no information or alert from the vendor third party payment engine or any other vendor of suspicious activity or a possible security breach until December 12, 2017.”
Unfortunately for the Village of Wellington, it seems they too are now caught up in Superion’s questionable patching and incident response practices. But at least this time, it was Superion that reached out to Wellington instead of waiting for a call from a bank fraud department:
“On June 6, 2018, the Village received a call from our vendor, Superion, notifying us of vulnerabilities in their software. The software problem was with the Click2Gov online payments for utility bills. Credit card information may have been taken during transactions.”
“The Village immediately shut down our payment connection to Superion and began working with them to determine if our resident’s information was compromised. The forensic analysis is continuing, security patches are being installed and new hardware and software are being installed to eliminate the breach. Even though Superion could not specifically confirm that our customer credit card data has been compromised, the Village decided to notify our customers as a precaution.”
There isn’t a lot publicly known about potential security issues with the Click2Gov solution. In taking a more detailed look at Superion’s website for any updates, there were none to be found for the Click2Gov software product. In fact, when looking on their website we were unable to find any links to security notices and when trying to find a dedicated security page (e.g. https://www.superion.com/security) we found nothing existed.
We then decided to reach out to Superion directly and email them at [email protected] as well as call their general enquiry and sales numbers. Unfortunately, both phone numbers gave the same automated message and then offered us to leave a voicemail.
As for a [email protected] mailbox, sadly but not unexpected, it bounced.
We then forward the message to their Media Inquiries address ([email protected]) to hopefully get some more information on the situation. If we receive a reply we will update this post.
What Comes Next?
The issue might affect quite a few more cities than initially expected. As we were conducting our investigation we attempted to determine how wide is the installation base of Click2Gov. Our results varied widely but what we found was that there appears to be between 600 to 6,000 installations of Click2Gov indexed (and potential thousands more depending how you look at it). Without spending much time digging, we quickly saw what appeared to be quite old versions of Click2Gov running.
Unfortunately, given what we have seen so far we anticipate seeing more breach reports coming to light thanks to the Click2Gov system. Superion and their clients are clearly struggling to wrap their hands around the problem and lock it down once and for all. In the meantime, any organization that is currently a Superion customer using Click2Gov should be on alert for suspicious activity. They should also consider reaching out to Superion for more information on the vulnerabilities that have been identified in Click2Gov, so that they can investigate whether they are exposed to the issue and implement patches or workarounds to mitigate the issue.
We suspect there will be more to this story and will update this post as we learn more. If you have any information please contact us!
Superion’s Press Contact replied to our email with the following:
Thank you for your email.
Protecting our customers and their clients’ data is of the utmost importance to Superion. Last year we reported that a limited number of on-premise clients had identified suspicious activity on their servers that are used to host Superion’s Clock2Gov product. Upon learning of the activity, we proactively notified all Click2Gov customers. Additionally, Superion launched an investigation and engaged a forensic investigator to assess what happened and determine appropriate remediation steps.
Throughout our investigation with the third-party forensic team, we have kept in direct contact with every Click2Gov customer to assist in the resolution of this issue, informing them of our findings via email, phone calls, and one-on-one working sessions. We assisted many customers with analyzing their Click2Gov environment and provided them with best-practice guidance to assist them in securing their servers and networks.
To date, Superion has deployed the necessary patch to our software and a related third-party component, and over 99% of these customers have applied these patches. At this time, we have no evidence showing that it is unsafe to make payments utilizing Click2Gov on hosted or secure on-premise networks with recommended patches and configurations. Superion does not control our customers’ networks, so we recommend citizens contact their municipality or county if they have any questions related to security.
Meanwhile, we continue to work closely with our customers to swiftly resolve and remediate this matter.