Dan Farmer, security researcher and creator of the SATAN vulnerability scanner, teamed up with HD Moore, chief research officer at Rapid7 and lead architect of the Metasploit penetration testing framework found 230,000 publicly accessible Out-Of-Band management interfaces on the Internet. Many of these systems were running software which dates back to 2001.
According to PCWorld, the Out-Of-Band (OOB) management interfaces expose servers to the Internet through microcontrollers embedded into the motherboard that run independently of the main OS and provide monitoring and administration functions. These microcontrollers are called Baseboard Management Controllers (BMCs). BMC’s are part of the Intelligent Platform Management Interface (IPMI), a standardized interface made up of a variety of sensors and controllers that allow administrators to manage servers remotely when they’re shut down or unresponsive, but are still connected to the power supply.
BMCs are embedded systems that have their own firmware—usually based on Linux. It’s an OS-agnostic and pervasive protocol. Initially developed by Intel (INTC), Dell (DELL), HP (HPQ), and other large vendor manufacturers, it was designed to help manage OOB or Lights-Out communication.
Pure IPMI is usually implemented as a network service that runs on UDP port 623 and can either piggyback on the server’s network port or may use a dedicated Ethernet port. Vendors take IPMI as a base and add on a variety of services like mail, SNMP, and Web GUIs, and then rebrand the new package:
- Dell has iDRAC,
- Hewlett Packard iLO,
- IBM (IBM) IMM2
It’s also used as the engine for higher-level protocols such as those put out by the DMTF (WBEM, CIM, etc.) the OpenStack Foundation, and others. IPMI is particularly popular for large-scale provisioning, roll-outs, remote troubleshooting, console access, and the like according to the research paper.
The parasitic BMC has near-complete control and oversight on of the server it rides upon, including its memory, networking, and storage media, and cannot be truly turned off; instead it runs continuously unless the power cord is completely pulled – an owner may only temporarily disable outside interaction unless you take a hammer to the motherboard.
Security researchers have warned in the past that most IPMI implementations suffer from architectural insecurities and other vulnerabilities that can be exploited to gain administrative access to BMCs. If attackers control the BMC they can mount attacks against the server’s OS as well as other servers from the same management group.
Dan Farmer stated in his recent paper Sold Down the River (PDF).
For over a decade major server manufacturers have harmed their customers by shipping servers that are vulnerable by default, with a management protocol that is insecure by design, and with little to no documentation about how to make things better … These vendors have not only gone out of their way to make their offerings difficult to understand or audit but also neglected to supply any substantial defense tools or helpful security controls
Mr. Farmer and Mr. Moore ran scans on the Internet in May 2014 and identified 230,000 publicly accessible BMCs. Deeper analysis of the at risk systems revealed:
- 46.8 percent of them were running IPMI version 1.5, which dates back to 2001,
- 53.2 percent were running IPMI version 2.0, which was released in 2004.
The researchers reported that nearly all the systems running IPMI v1.5 were configured so that all accounts could be logged into without authentication, “… you can login to pretty much any older IPMI system without an account or a password.” Mr.. Farmer explains this set-up can grant an attacker privileged access, “… in most cases they grant administrative access, and even when they don’t the mere ability to execute any kind of commands without authentication is a bad thing.”
The team found that IPMI v.2.0, which includes cryptographic protection has it own security issues. For example, the first cipher option, known cipher zero, provides no authentication, integrity or confidentiality protection, Farmer said. A valid user name is required for logging in, without a password. The researcher found that around 60 percent of the publicly accessible BMCs running IPMI version 2 had this vulnerability.
Another serious issue introduced by IPMI 2.0 stems from its RAKP key-exchange protocol that’s used when negotiating secure connections. The protocol allows an anonymous user to obtain password hashes associated with any accounts on the BMC, as long as the account names are known.
“This is an astonishingly bad design, because it allows an attacker to grab your password’s hash and do offline password cracking with as many resources as desired to throw at the problem,” Farmer said.
The analysis showed that 83 percent of the identified BMCs were vulnerable to this issue and a test with brute-force password guessing application John the Ripper, using a modest 4.7 million-word dictionary successfully cracked 30% of the BMC passwords.
Farmer calculated that between 72.8 and 92.5 percent, depending on password cracking success rate, of BMCs running IPMI 2.0 had authentication issues and were vulnerable to unauthorized access.
“While a quarter of a million BMCs is only a tiny sliver of the total computing power in the world, it’s still an important indicator as a kind of canary in the coalmine,” because BMCs that are behind corporate firewalls share the same issues, Farmer said. “While management systems are often not directly assailable from the outside they’re often left open once the outer thin hard candy shell of an organization is breached.”
The research paper includes some recommendations for server administrators on how to mitigate some of the identified issues and better secure their BMCs, but the researcher concludes that ultimately the problem of insecure IPMI implementations will linger on for a long time. Mr. Farmer concludes with a rant:
Many of these problems would have been easy to fix if the IPMI protocol had undergone a serious security review or if the developers of modern BMCs had spent a little more effort in hardening their products and giving their customers the tools to secure their servers … At this point, it is far too late to effect meaningful change. The sheer number of servers that include a vulnerable BMC will guarantee that IPMI vulnerabilities and insecure configurations will continue to be a problem for years to come.
They told us so, about a year ago.
Defense in-depth, block UDP port 623 at the perimeter – yes all of them, on the end-points, you are using personal firewalls?
Disable or remove the default vendor user names and pick a strong UID and PWD
Least privilege, the researchers warn that anyone who has administrative privileges on a BMC’s server has administrative control over it and may disable or enable IPMI, add or remove accounts, change the IP address, etc., etc.–all without any authentication to the BMC.
Ralph Bach has been in IT for fifteen years and has blogged from his Bach Seat about IT, careers and anything else that catches his attention since 2005. You can follow me at Facebook and Twitter. Email the Bach Seat here.