gwbas1c
To summarize for people who TLDR: 14 freezers failed at the same time. They are all internet-controlled, and failed at the same time as a disclosure about a vulnerability. They all failed by turning on the defrost cycle and heating food.
Regardless if this was a hack or a bug, the bigger lesson is that overcomplicated systems fail in catastrophic ways. Why do military commissaries need remote-controlled freezers? It seems like a very fragile, and needless, way to run a freezer.
---
But, there are some options that the author didn't consider:
1: This could be a quickly applied patch that failed.
2: This could be a "script kiddie" hack from someone who isn't a government actor.
I'm less onboard with a state actor. Generally, when a state actor has hacked something, they don't want the victim to know. In this case, if it was a state actor, I would anticipate that they would make a single freezer fail in a way that they could verify using something like a hacked video camera or otherwise by watching public social media feeds. IMO: A state actor would only "make sense" if they knew the hole was closing soon and they don't care if they're discovered, perhaps because their operation is winding down.
quickthrowmanreply
These aren’t your typical refrigerator or freezer, these facilities have walk-ins or purpose built cold storage with multiple evaporators and condensing units. A building automation system is pretty standard for most buildings above a certain size, and monitoring and controlling the refrigeration is usually a part of it.
Unfortunately, I would wager that all BAS software is full of flaws and holes, allowing access to it for the public internet seems like a bad idea. I need to be on my company VPN to access our locally hosted BAS front end (which I have authorized access for) which seems like the bare minimum security.
stackghostreply
>Generally, when a state actor has hacked something, they don't want the victim to know
Could be the Iranians, or someone aligned, conducting anti-morale operations. Could be the start of a series of small but annoying failures.
TeMPOraLreply
Or could be a rouge LLM in one of the big labs, that accidentally self-prompt-injected itself with the title of that vulnerability research paper.
stackghostreply
I suppose.
But if I was an Iranian operative with instructions to damage the morale of the American war machine, I'd absolutely do a series of things like this.
It doesn't really harm anyone, but enough similar events and the families of servicemembers start to feel unsafe, which is psychologically very different than the servicemembers themselves feeling unsafe.
It's probably more benign than that, most likely firmware or a central controller failure, but still
elictronicreply
Iran is actively looking for ways to attack back against the United States especially against military targets without actually escalating the situation.
I'm sure some script kiddie broke into a government network, hacked an industrial process, and forced a limited supply piece of equipment into a failure mode that takes some thought and is more unique as an attack vector. It's just like buying hacks for CS source right?
peterabbitcook
A couple years ago I worked on a service that had to communicate with a Siemens S7-1500 PLC. Based on my experience with that project, none of what I’ve read recently about unsecured industrial PLCs is surprising.
I opened Siemens TIA Portal and PLCSIM for the first time and thought “wow, I didn’t think the Windows 95 GUI library was still supported.” None of the PLC contractors we had hired knew how to enable TLS on the thing (user/pass eg admin/admin was their usual). Anecdote: I once spent hours reading the docs and clicking around trying to get it to accept an SSL certificate signed by a real CA and it wouldn’t go, but it accepted one I self-signed in openssl.
In all fairness, the people who are experts in the field of Siemens PLC programming are usually mechanical-ish engineers and security is not in their skill set or on their mind.
sidewndr46reply
Isn't this the industry expectation in that kind of equipment? If it was signed by a real CA the cert. could expire and render the equipment unable to communicate.
stephbookreply
also you can't pin the user/pw to the machine with a note, because someone might need remote access. better stick with admin/admin
peterabbitcookreply
That’s a tough question. If your PLC is on an airgapped LAN, admin/admin is not great security hygiene but you’ve reduced most of the risk by airgapping.
On my project the service I wrote was doing bidirectional communication with the PLC over OPCUA. The server running this pod was connected to the internet, so it was critical to have proper TLS for the OPCUA client/server. Rotating LetsEncrypt certs on the system every 45 days is a lot of toil, but using a self-signed cert that expires in 2040 from some dev laptop doesn’t pass muster in most organizations either.
That’s just the OPCUA path.. In these projects You also typically see WinCC HMIs that can talk to the PLC from anywhere on the network without TLS. And also SIMATIC Web Server pages - minimally secured by default, and the amount of info you can grok about the system in Chrome Dev Tools is troubling.
dylan604reply
> Rotating LetsEncrypt certs on the system every 45 days is a lot of toil
What is unique about your system using LetsEncrypt that you can't automate certbot to handle this task as it was designed and intended to be done?
sidewndr46reply
I'm not sure if you're speaking from personal experience, but most I've interacted with don't have to worry about the self-signed vs. LetsEncrypt debate. They just don't do it. Also there would be no way to do LetsEncrypt as the system is air gapped.
gopher_spacereply
> admin/admin is not great security hygiene but you’ve reduced most of the risk by airgapping.
admin/admin is on a post-it stuck to the monitor because everyone we hire is perfectly capable of rooting a machine they have physical access to.
tgsovlerkhgselreply
My guess would be something like the CA using some feature that was newer than what the equipment would support (e.g. ECC signatures but the equipment only supporting RSA), not an intentional "no real CAs" decision.
katzenversteherreply
Most factories I know do not allow their PLC be accessed from the internet. They are usually on a separate Network. However, the "engineering" station (the computer running e.g. TIA Portal) sometimes is.
The PLC engineers I had contact with usually had an electrical engineering background. That's why they like PLCs in the first place with the ladder logic programming languages, grafcet and if they feel fancy a bit of structured text (assembly like) or structured control language (pascal like). They indeed did not know much about software security but a great deal about machine safety.
A real security nightmare are older OPC servers (OPC-DA) which is super reliant on DCOM. OPC is quite important to connect the PLCs to SCADA systems or 3rd party devices.
lowbloodsugarreply
>They are usually on a separate Network.
Then someone plugs in a cable because boss wants something "over there" and there's already a network that runs "over there". Or optimizes to a smart switch with vlans, and then someone else optimizes to a single vlan. It's not hard to not give a shit, or not understand, network security.
lenerdenatorreply
That someone can be brought into an office and shown a small diagram of the approved network topology. Then they can be shown a small diagram of the current network topology. Next, they can be asked if they're the same. If they're not, they can finally be asked if they're aware that deviating from the approved network topology without consulting infosec is grounds for termination of their employment.
jordanbreply
My mind was blown when I realized that the way tftp works is that as the machine is booting it asks the network if anyone has some software for it to run.
Joker_vDreply
Well, what else can it do, really? It has to boot with pretty much zero knowledge about the external world (maybe except asking the user for the current date and time). Sure, you can hardcode an outdated list of CAs (it's always outdated because the system can be booted 10/20/100 years after it was made) in but that just opens you to unexpected obsolescence, and you usually can't put too much stuff in the bootloader anyway.
Not really dissimilar from the human upbringing: leave a baby with "bad" guardians, and it will grow up corrupted. That's a feature, not a bug: if you knew what behaviour exactly you wanted (other than "whatever Simon says"), you would just bake it in in the first place, right?
Dylan16807reply
If we pretend we're revising TFTP boot in 1995, let's have it get up to 20 boot options from the server and their md5 hashes, and if it's not set to auto it waits for the user to pick one. It then verifies the hash as it downloads. Also it uses TCP for the download.
CobaltFire
As someone who spent over 20 years active duty, and spent a ton of my career in the IT, security, etc. side of the house:
Unlikely to be a hack, more likely to be a misconfiguration or update sent incorrectly.
That said, the timing of the disclosure and the issue are rather concerning.
Regarding the highest value targets to hit with an attack like this, you would want to target Guam, Hawai'i, and other isolated overseas locations where this would have ripple effects in the local economy. Guam specifically would cause catastrophic supply shortages, since DeCA probably supplies around 50% of the groceries on that island (that's a WAG based on my time there).
ericmayreply
Generally agree with your assessment, but in the case of Guam or other more remote installations if there were catastrophic issues we'd just airlift food in. Costly but certainly manageable.
Hawaii I'm not sure why that would be an issue unless the whole island was attacked or shut down. Even if the on-base shops were hacked you could just go shop at Wal-Mart or Costco or any number of other locations on the islands. If there was an extended issue then the commander could authorize meal stipends as they do for some units today and then you would just go buy food off-base. Ideal? No. Manageable? Very much.
scheme271reply
More to the point for Guam and similar locations, canned and non-perishable food can are probably around and more can be airlifted in to get people through it. It was pretty much the standard back in the day.
avs733reply
The diet in Guam is already heavily dependent on shelf stable foods. I know this from personal experience but theres a surprising (to me) amount of research on it as well (c.v., https://www.guampedia.com/health-consequences-of-modern-diet...). However, it is worth noting that food on a military installation =/= food nearby, for many obvious and non obsvious reasons.
cyanydeezreply
With the current admin, assumptions about capabilities are just wrong.
One of the reasons for consitency above all else is to use the power of history.
Ahistorical is now the defacto standard. All asdumptions about America power are toilet paper.
nwatsonreply
Did they airlift better supplies to a floating aircraft carrier somewhere in the northern Indian Ocean recently? No.
nkriscreply
Surely airlifting supplies to a carrier in a war zone is not totally comparable to airlifting supplies to a base in a US territory?
nwallinreply
Aircraft carriers are not particularly well suited to be supplied by cargo aircraft.
The largest cargo aircraft with the strengthened landing gear, arresting hook, and catapult attachment used to be the C-2 Greyhound. It had a range of approximately 1,500 miles with a 10,000 pound cargo. It was retired on July 28th, 2026. The C-2 has been replaced by the CMV-22B Osprey. It has a range of only 1,150 miles and a more limited cargo capacity of only 6,000 pounds. 6,000 pounds isn't nothing, but it's not a lot either.
There's not a lot of stuff within 1,150mi of the Arabian Sea. Your two options are one of the US bases in the southern Persian Gulf (Kuwait is too far) and Diego Garcia. Neither are ideal. In both cases you're doing a transfer out in the field.
On the other hand, the C-5 Galaxy can carry 120,000 pounds to a range of 4,800 miles. You can fill it with food in the continental US, fly to Hickam AFB in Hawaii, refuel, and fly the rest of the way to Guam. On the return leg, an unladen C-5 can fly directly to the 22nd Airlift Squadron's home base California without a layover.
The C-17B Globemaster is smaller than the C-5, although it's nearly as capable. With a 160,000 pound cargo, it has a range of 3,200 miles, (the C-17A has a shorter range) which is just short of the 3,800 miles to get from Hawaii to Guam. I don't know how much cargo it's capable of carrying 3,800 miles, but on the order of 100,000 pounds is a reasonable guess. It is also capable of an empty flight back to the US without a layover.
psunavy03reply
You're just revealing your ignorance of Navy supply chains with this comment, and I say that as a veteran who has no support for the ongoing shitshow, only sympathy for my former colleagues trapped underway.
TZubirireply
>Unlikely to be a hack, more likely to be a misconfiguration or update sent incorrectly.
Are these materially different? Assuming that there wasn't a literal command to turn off all freezers, from an organizational, non criminal perspective, to the organization the damage will be the same, the root cause will be a bug (whether exploited by chance or malice), and the fix will be the same(fixing the bug).
gavinsyanceyreply
> Are these materially different?
A hack implies an adversary, and intentional targeting.
TZubirireply
But is it materially different? Or is it a difference in quantitative increase of likelihood of the exploit?
The thesis "everything hackable" will be hacked, and with the advent of hacking LLMs, the boundary between criminal hacking and civil hacking is being blurred, it's no longer attributable to a criminal intent, hacking commands are just something that happens through no criminal intent.
Practically, the hackers, whether human or syntehtic, are a catalyst, they accelerate the rate at which a bug that causes damage will manifest, and perhaps the magnitude of the manifestation, but the essence of the negative manifestation of that bug is the same. Whether it was triggered by a solar ray flipbit or by a rogue LLM or by a russian hacker, the essence of the issue is the same.
segmondyreply
I always wonder by folks with some "inside" knowledge like you will then come out to share more details. Why? I understand that there's no security through obscurity, but I don't think that the details to get your point across matters. Loose lips sink ships.
munificentreply
We are a very social species who likely evolved to feel good when we share juicy information.
CobaltFirereply
Nothing I shared is, in any way, even close to something I am concerned with sharing.
This type of wargaming scenario is literally stuff they give to brand new kids out of high school to get them thinking. It's not anything close to "loose lips sink ships."
0xWTF
This is eerily suggestive of a vulnerability Hank Paulson hinted at in his 2014 book "Dealing with China"
"Every nonelectric cooler comes with 25 years of free real-time monitoring. On a visit to the company in the spring of 2012, I watched as technicians in Broad Air’s space-age control room checked on the performance of its units in locations as diverse as the Adolfo Suárez Madrid–Barajas Airport in Spain; Qualcomm headquarters in San Diego, California; and Fort Stewart, the U.S. Army base in Georgia [emphasis added].
"Zhang says that 80 percent of his clients are repeaters. “If you bring long-term benefits for your clients, they will choose you.”"
Terr_reply
America's tech-sector has a similar problem, which--until recently--was tempered by the idea that it was a dependable and predictable ally to most of its customers.
Not just in the sense of secretive kill-switches, but "US government commands you to turn over this encryption key and you're not allowed to say you did so" stuff.
vkoureply
> "US government commands you to turn over this encryption key and you're not allowed to say you did so"
That barely even registers given the scale of more modern escalations against its 'allies'. Escalations like "US government threatens that you aren't a real country and that you can't defend yourself from it."