{"id":69826,"date":"2014-06-12T16:06:58","date_gmt":"2014-06-12T20:06:58","guid":{"rendered":"http:\/\/rbach.net\/blog\/index.php\/"},"modified":"2021-08-25T21:14:59","modified_gmt":"2021-08-26T01:14:59","slug":"server-management-security-hole","status":"publish","type":"post","link":"https:\/\/rbach.net\/index.php\/server-management-security-hole\/","title":{"rendered":"Server Management Security Hole"},"content":{"rendered":"<p><a href=\"https:\/\/web.archive.org\/web\/20210421074735\/https:\/\/erpfm.com\/cloud-erp-and-benefits\/\" target=\"_blank\" rel=\"noopener noreferrer\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" class=\"alignleft wp-image-107195\" title=\"Server Management Security Hole\" src=\"https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/remote_work-1-e1570225435981-132x150.jpg?resize=96%2C110&#038;ssl=1\" alt=\"Server Management Security Hole\" width=\"96\" height=\"110\" srcset=\"https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/remote_work-1-e1570225435981.jpg?resize=132%2C150&amp;ssl=1 132w, https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/remote_work-1-e1570225435981.jpg?resize=66%2C75&amp;ssl=1 66w, https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/remote_work-1-e1570225435981.jpg?resize=768%2C876&amp;ssl=1 768w, https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/remote_work-1-e1570225435981.jpg?resize=898%2C1024&amp;ssl=1 898w, https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/remote_work-1-e1570225435981.jpg?w=1368&amp;ssl=1 1368w, https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/remote_work-1-e1570225435981.jpg?w=960&amp;ssl=1 960w\" sizes=\"auto, (max-width: 96px) 100vw, 96px\" \/><\/a><a href=\"http:\/\/en.wikipedia.org\/wiki\/Dan_Farmer\" target=\"_blank\" rel=\"noopener noreferrer\">Dan Farmer<\/a>, security researcher and creator of the <a title=\"SATAN vulnerability scanner\" href=\"https:\/\/www.cerias.purdue.edu\/site\/about\/history\/coast\/satan.php\" target=\"_blank\" rel=\"noopener noreferrer\">SATAN vulnerability scanner,<\/a> teamed up with <a title=\"HD Moore\" href=\"https:\/\/community.rapid7.com\/people\/hdmoore\" target=\"_blank\" rel=\"homepage noopener noreferrer\">HD Moore<\/a>, chief research officer at <a title=\"Rapid7\" href=\"http:\/\/www.rapid7.com\/\" target=\"_blank\" rel=\"noopener noreferrer\">Rapid7<\/a> and lead architect of the <a title=\"Metasploit\" href=\"https:\/\/community.rapid7.com\/community\/metasploit\/blog\" target=\"_blank\" rel=\"noopener noreferrer\">Metasploit<\/a> penetration testing framework found <strong>230,000 publicly accessible <\/strong><a title=\"Out-of-band management\" href=\"http:\/\/en.wikipedia.org\/wiki\/Out-of-band_management\" target=\"_blank\" rel=\"noopener wikipedia noreferrer\">Out-Of-Band management<\/a><strong> interfaces on the Internet<\/strong>. Many of these systems were running software that dates back to 2001.<\/p>\n<h3>Out-Of-Band server management<\/h3>\n<p><a href=\"https:\/\/web.archive.org\/web\/20201111233541\/http:\/\/www.makeuseof.com\/tag\/is-the-microsoft-fix-it-service-really-any-good-si-x2\/\" target=\"_blank\" rel=\"noopener noreferrer\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" class=\"alignright wp-image-107196\" title=\"Out-Of-Band (OOB) management\" src=\"https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/microsoft_fix-it-logo.png?resize=100%2C100&#038;ssl=1\" alt=\"Out-Of-Band (OOB) management\" width=\"100\" height=\"100\" srcset=\"https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/microsoft_fix-it-logo.png?resize=150%2C150&amp;ssl=1 150w, https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/microsoft_fix-it-logo.png?resize=75%2C75&amp;ssl=1 75w, https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/microsoft_fix-it-logo.png?w=300&amp;ssl=1 300w\" sizes=\"auto, (max-width: 100px) 100vw, 100px\" \/><\/a>According to <a title=\"Many servers expose insecure out-of-band management interfaces to the Internet\" href=\"http:\/\/www.pcworld.com\/article\/2361040\/many-servers-expose-insecure-outofband-management-interfaces-to-the-internet.html\" target=\"_blank\" rel=\"noopener noreferrer\"><em>PCWorld<\/em><\/a>, the <strong>Out-Of-Band (OOB) management<\/strong> interfaces expose servers to the Internet through microcontrollers embedded into the motherboard that <strong>run independently of the main OS<\/strong> and provide monitoring and administration functions. These\u00a0microcontrollers are called\u00a0<strong>Baseboard Management Controllers<\/strong> (BMCs). BMC&#8217;s are part of the <a href=\"http:\/\/en.wikipedia.org\/wiki\/Intelligent_Platform_Management_Interface\" target=\"_blank\" rel=\"noopener noreferrer\"><strong>Intelligent Platform Management Interface<\/strong> (IPMI)<\/a>, a standardized interface made up of a variety of sensors and controllers that allow <strong>administrators to manage servers remotely<\/strong> when they\u2019re shut down or unresponsive, but are still connected to the power supply.<\/p>\n<p>BMCs are <strong>embedded systems<\/strong> that have their own firmware\u2014usually based on <strong><a title=\"Linux\" href=\"https:\/\/web.archive.org\/web\/20151223033023\/http:\/\/www.intenseschool.com:80\/boot_camp\/linux\" target=\"_blank\" rel=\"intenseschoollinux noopener noreferrer\">Linux<\/a><\/strong>. It\u2019s an OS-agnostic and pervasive protocol. Initially developed by <a title=\"Intel\" href=\"https:\/\/www.intel.com\/\" target=\"_blank\" rel=\"noopener noreferrer\">Intel<\/a> (<a title=\"HASDAQ : INTC\" href=\"https:\/\/www.tradingview.com\/symbols\/NASDAQ-INTC\/\" target=\"_blank\" rel=\"noopener noreferrer\">INTC<\/a>), <a href=\"http:\/\/www.dell.com\/\" target=\"_blank\" rel=\"noopener noreferrer\">Dell<\/a> (<a href=\"https:\/\/www.tradingview.com\/symbols\/NYSE-DELL\/\" target=\"_blank\" rel=\"noopener noreferrer\">DELL<\/a>), <a title=\"HP\" href=\"http:\/\/www.hp.com\/\" target=\"_blank\" rel=\"noopener noreferrer\">HP<\/a> (<a title=\"NYSE : HPQ\" href=\"https:\/\/www.nyse.com\/quote\/XNYS:HPQ\" target=\"_blank\" rel=\"noopener noreferrer\">HPQ<\/a>), and other large equipment manufacturers. It was designed to help manage OOB or Lights-Out communication.<\/p>\n<h3>Rebranded by OEM manufacturers<\/h3>\n<p><a href=\"https:\/\/web.archive.org\/web\/20201111233541\/http:\/\/www.makeuseof.com\/tag\/is-the-microsoft-fix-it-service-really-any-good-si-x2\/\" target=\"_blank\" rel=\"noopener noreferrer\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" class=\"alignright wp-image-107198 size-medium\" title=\"Lights-Out communication\" src=\"https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/server-6400r2.jpg?resize=150%2C69&#038;ssl=1\" alt=\"Lights-Out communication\" width=\"150\" height=\"69\" srcset=\"https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/server-6400r2.jpg?resize=150%2C69&amp;ssl=1 150w, https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/server-6400r2.jpg?resize=75%2C35&amp;ssl=1 75w, https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/server-6400r2.jpg?w=392&amp;ssl=1 392w\" sizes=\"auto, (max-width: 150px) 100vw, 150px\" \/><\/a>Pure IPMI is usually implemented as a network service that runs on <strong>UDP port 623. <\/strong>It can either piggyback on the server\u2019s network port or may use a dedicated Ethernet port. Vendors take IPMI as a base and add on a variety of services like mail, SNMP, and Web GUIs, and then rebrand the new package:<\/p>\n<ul>\n<li>Dell has iDRAC,<\/li>\n<li>Hewlett Packard iLO,<\/li>\n<li><a title=\"IBM\" href=\"http:\/\/www.ibm.com\/\" target=\"_blank\" rel=\"noopener noreferrer\">IBM<\/a> (<a title=\"NYSE : IBM\" href=\"https:\/\/www.nyse.com\/quote\/XNYS:IBM\" target=\"_blank\" rel=\"noopener noreferrer\">IBM<\/a>)\u00a0IMM2<\/li>\n<\/ul>\n<p>It\u2019s also used as the engine for higher-level protocols. Some of the protocols are put out by the DMTF (WBEM, CIM, etc.) the <a title=\"OpenStack Foundation\" href=\"https:\/\/www.openstack.org\/foundation\/\" target=\"_blank\" rel=\"noopener noreferrer\">OpenStack Foundation<\/a>, and others. IPMI is particularly popular for large-scale provisioning, roll-outs, remote troubleshooting, and console access according to the research paper.<\/p>\n<h3>Parasitic oversight<\/h3>\n<p><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" class=\"alignright wp-image-107202\" title=\"complete control and oversight on of the server\" src=\"https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/man-with-clipboard-1.jpg?resize=76%2C110&#038;ssl=1\" alt=\"complete control and oversight on of the server\" width=\"76\" height=\"110\" srcset=\"https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/man-with-clipboard-1.jpg?resize=104%2C150&amp;ssl=1 104w, https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/man-with-clipboard-1.jpg?resize=52%2C75&amp;ssl=1 52w, https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/man-with-clipboard-1.jpg?w=555&amp;ssl=1 555w\" sizes=\"auto, (max-width: 76px) 100vw, 76px\" \/>The parasitic BMC has <strong>near-complete control and oversight of the server it rides upon. <\/strong>It can control the server&#8217;s including its memory, networking, and storage media. It can not be truly turned off. Instead, it runs continuously unless the power cord is completely pulled. An owner may only temporarily disable outside interaction unless you take a hammer to the motherboard.<\/p>\n<p>Security researchers have warned in the past that most IPMI implementations suffer from architectural insecurities and other vulnerabilities\/ These can be exploited to gain administrative access to BMCs. <strong>If attackers control the BMC they can mount attacks against the server\u2019s OS<\/strong> as well as other servers from the same management group.<\/p>\n<p>Dan Farmer stated in his recent paper <a title=\"Sold Down the River\" href=\"https:\/\/web.archive.org\/web\/20221014070621\/http:\/\/fish2.com\/ipmi\/river.pdf\" target=\"_blank\" rel=\"noopener noreferrer\"><em>Sold Down the River<\/em><\/a> (PDF).<\/p>\n<p style=\"padding-left: 30px; text-align: justify;\"><em>For over a decade major server manufacturers have harmed their customers by shipping servers that are vulnerable by default, with a management protocol that is insecure by design, and with little to no documentation about how to make things better &#8230; These vendors have not only gone out of their way to make their offerings difficult to understand or audit but also neglected to supply any substantial defense tools or helpful security controls.<\/em><\/p>\n<h3>Old BMC software<\/h3>\n<p><a href=\"https:\/\/www.yeastar.com\/remote-management\/\" target=\"_blank\" rel=\"noopener noreferrer\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" class=\"alignright wp-image-107193\" title=\"Remote management\" src=\"https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/remote_mgt.png?resize=137%2C95&#038;ssl=1\" alt=\"Remote management\" width=\"137\" height=\"95\" srcset=\"https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/remote_mgt.png?resize=150%2C104&amp;ssl=1 150w, https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/remote_mgt.png?resize=75%2C52&amp;ssl=1 75w, https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/remote_mgt.png?w=382&amp;ssl=1 382w\" sizes=\"auto, (max-width: 137px) 100vw, 137px\" \/><\/a>Mr. Farmer and Mr. Moore ran scans on the Internet in May 2014 and identified 230,000 publicly accessible BMCs. A deeper analysis of the at-risk systems revealed:<\/p>\n<ul>\n<li>46.8% of them were running IPMI version 1.5, which dates back to 2001,<\/li>\n<li>53.2% were running IPMI version 2.0, which was released in 2004.<\/li>\n<\/ul>\n<p>The researchers reported that nearly all the systems running IPMI v1.5 were configured so that all accounts could be <strong>logged into without authentication. <\/strong>\u201c<em>&#8230; you can login to pretty much any older IPMI system without an account or a password<\/em>.\u201d Mr. Farmer explains this set-up can grant an attacker privileged access, &#8220;<em>&#8230; in most cases, they grant administrative access, and even when they don\u2019t the mere ability to execute any kind of commands without authentication is a bad thing.<\/em>&#8221;<\/p>\n<p><a href=\"http:\/\/rollonfriday.com\/TheNews\/EuropeNews\/tabid\/58\/Id\/2427\/fromTab\/36\/currentIndex\/0\/Default.aspx\" target=\"_blank\" rel=\"noopener noreferrer\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" class=\"alignright wp-image-107204 size-medium\" title=\"architectural insecurities that can be exploited\" src=\"https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/broken_server.jpg?resize=150%2C98&#038;ssl=1\" alt=\"architectural insecurities that can be exploited\" width=\"150\" height=\"98\" srcset=\"https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/broken_server.jpg?w=150&amp;ssl=1 150w, https:\/\/i0.wp.com\/rbach.net\/wp-content\/uploads\/broken_server.jpg?resize=75%2C49&amp;ssl=1 75w\" sizes=\"auto, (max-width: 150px) 100vw, 150px\" \/><\/a>The team found that IPMI v.2.0, which includes cryptographic protection has its own security issues. For example, the first cipher option, known as cipher zero, provides <strong>no authentication, integrity, or confidentiality protection<\/strong>, Farmer said. A valid user name is required for logging in, without a password. The researcher found that around 60% of the publicly accessible BMCs running IPMI version 2 had this vulnerability.<\/p>\n<h3>Server management issues in IPMI 2.0<\/h3>\n<p>Another serious issue introduced by IPMI 2.0 stems from its RAKP key-exchange protocol that\u2019s used when negotiating secure connections. The protocol allows an anonymous user to obtain password hashes associated with any accounts on the BMC, as long as the account names are known.<\/p>\n<p style=\"text-align: justify; padding-left: 30px;\"><em>&#8220;This is an astonishingly bad design, because it allows an attacker to grab your password\u2019s hash and do offline password cracking with as many resources as desired to throw at the problem,\u201d Farmer said.<\/em><\/p>\n<p>The analysis showed that 83% of the identified BMCs were vulnerable to this issue. A test with brute-force password guessing application <a title=\"john the ripper\" href=\"http:\/\/www.openwall.com\/john\/\" target=\"_blank\" rel=\"noopener noreferrer\">John the Ripper<\/a>, using a modest 4.7 million-word dictionary successfully cracked 30% of the BMC passwords.\u00a0Farmer calculated that between 72.8 and 92.5% depending on password cracking success rate, of BMCs running IPMI 2.0 had authentication issues and were vulnerable to unauthorized access.<\/p>\n<h3>Canary in the coal mine<\/h3>\n<p>\u201c<em>While a quarter of a million BMCs is only a tiny sliver of the total computing power in the world, it\u2019s still an important indicator as a kind of canary in the coal mine,<\/em>\u201d Mr. Farmer warns. He predicts that BMCs behind corporate firewalls share the same issues. He said. \u201c<em>While management systems are often not directly assailable from the outside they\u2019re often left open once the outer thin hard candy shell of an organization is breached.<\/em>\u201d<\/p>\n<p>The research paper includes recommendations for server administrators on how to mitigate some of the identified issues and better secure their BMCs. But the researcher concludes that ultimately the problem of insecure IPMI implementations will linger on for a long time. Mr. Farmer concludes with a rant:<\/p>\n<p style=\"text-align: justify; padding-left: 30px;\"><em>Many of these problems would have been easy to fix if the IPMI protocol had undergone a serious security review or if the developers of modern BMCs had spent a little more effort in hardening their products and giving their customers the tools to secure their servers &#8230; At this point, it is far too late to effect meaningful change. The sheer number of servers that include a vulnerable BMC will guarantee that IPMI vulnerabilities and insecure configurations will continue to be a problem for years to come.<\/em><\/p>\n<p><em><strong>rb-<\/strong><\/em><br \/>\n<em> They <a title=\" A Penetration Tester's Guide to IPMI and BMCs\" href=\"https:\/\/community.rapid7.com\/community\/metasploit\/blog\/2013\/07\/02\/a-penetration-testers-guide-to-ipmi\" target=\"_blank\" rel=\"noopener noreferrer\">told us so<\/a>, about a year ago.<\/em><\/p>\n<p><em><strong>Defense-in-depth<\/strong>, block <strong>UDP port 623<\/strong> at the perimeter &#8211; yes all of them, on the end-points, you are using personal firewalls?<\/em><\/p>\n<p><em><strong>Disable or remove the default vendor user names<\/strong> and pick a strong UID and PWD<\/em><\/p>\n<p><em><strong>Least privilege<\/strong>, the researchers warn that anyone who has administrative privileges on a BMC\u2019s server has administrative control over it and may disable or enable IPMI, add or remove accounts, change the IP address, etc., etc.\u2013all without any authentication to the BMC.<\/em><\/p>\n<h6>Related articles<\/h6>\n<ul>\n<li><a title=\"Intelligent Platform Management Interface\" href=\"https:\/\/www.intel.com\/content\/www\/us\/en\/servers\/ipmi\/ipmi-home.html\" target=\"_blank\" rel=\"noopener noreferrer\">Intelligent Platform Management Interface<\/a> (<a title=\"www.intel.com\" href=\"https:\/\/www.intel.com\/\" target=\"_blank\" rel=\"noopener noreferrer\">www.intel.com<\/a>)<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><em><a title=\"Ralph Bach\" href=\"https:\/\/rbach.net\/index.php\/new-resume\/\" target=\"_blank\" rel=\"noopener noreferrer\">Ralph Bach<\/a>\u00a0has been in IT long enough to know better and has blogged from his\u00a0<a title=\"Bach Seat\" href=\"https:\/\/rbach.net\/\" target=\"_blank\" rel=\"noopener noreferrer\">Bach Seat<\/a> about IT, careers, and anything else that catches his attention since 2005. You can follow him on <a class=\"broken_link\" href=\"http:\/\/www.linkedin.com\/in\/rb48334\" target=\"_blank\" rel=\"noopener noreferrer nofollow\">LinkedIn<\/a>,\u00a0<a href=\"https:\/\/www.facebook.com\/ralph.bach.14\" target=\"_blank\" rel=\"noopener noreferrer\">Facebook<\/a>,\u00a0and\u00a0<a href=\"https:\/\/twitter.com\/rbach48334\" target=\"_blank\" rel=\"noopener noreferrer\">Twitter<\/a>. Email the Bach Seat\u00a0<a href=\"mailto:\/\/bach.seat@gmail.com\" target=\"_blank\" rel=\"noopener noreferrer\">here<\/a>.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>230,000+ servers with IPMI OOB server management open to the web &#038; nearly half run monitoring software from 2001 didn&#8217;t need a login to take over server HPQ Dell.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[2292,2142,2140,216,2141,168,904,2145,2144,2143,780,2146,4,106],"class_list":["post-69826","post","type-post","status-publish","format-standard","hentry","category-security","tag-2292","tag-bmc","tag-dan-farmer","tag-dell","tag-hd-moore","tag-hp","tag-hpq","tag-idrac","tag-ilo","tag-ipmi","tag-linux","tag-oob","tag-security","tag-servers"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/rbach.net\/index.php\/wp-json\/wp\/v2\/posts\/69826","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rbach.net\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rbach.net\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rbach.net\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rbach.net\/index.php\/wp-json\/wp\/v2\/comments?post=69826"}],"version-history":[{"count":11,"href":"https:\/\/rbach.net\/index.php\/wp-json\/wp\/v2\/posts\/69826\/revisions"}],"predecessor-version":[{"id":120291,"href":"https:\/\/rbach.net\/index.php\/wp-json\/wp\/v2\/posts\/69826\/revisions\/120291"}],"wp:attachment":[{"href":"https:\/\/rbach.net\/index.php\/wp-json\/wp\/v2\/media?parent=69826"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rbach.net\/index.php\/wp-json\/wp\/v2\/categories?post=69826"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rbach.net\/index.php\/wp-json\/wp\/v2\/tags?post=69826"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}