Admin Network__Security_-_Issue_70_2022

ADMIN FREE Apache Mahout Defense by Design Network & Security DVD ADMIN Network & Security Defense by ISSUE70 Design • Extended detection and response • Ransomware contingency plans • DNS cyber defense APACHE MAHOUT Distributed linear algebra framework Azure AD with Conditional Access Jaeger Troubleshoot cloud-native applications Prometheus Anomaly Detector AD Sites and Services Puppet Bolt Weka Replication between sites Automate admin tasks Statistics and ML Foundries.io Teleport Develop IoT apps for devices Centrally manage logins WWW.ADMIN-MAGAZINE.COM



Welcome to ADMIN WELCOME Lead Image © Oleksandr Omelchenko, 123RF.com The Great Stay Put The IT field is either feast or famine, depending on your perspective, but one thing is certain: No matter where you decide to land, you always have value. A few months ago, I wrote this column describing “The Great Resignation.” Now, I think I must write this one as “The Great Stay Put,” because things have changed so rapidly since then. I had real wide-eyed optimism about the IT job market, but all that has changed. Now, just as was seen in the early 2000s, you might be lucky to have a job in six months. I hope that’s not the case, but you need to prepare yourself. One way to prepare is to change to a safer position, and by “safer,” I mean one that is essential to business operations. The other way is to stay where you are and ride this thing out. There’s no way to predict accurately what’s going to happen six months from now, but that doesn’t mean you have to sit around and just wait for it to happen to you. You can act now to be sure that you decrease any negative effect from a downturn in the economy. Whether you decide to stay or leave your current job, you must position yourself to show that your presence is im- portant to the ongoing health of the business. The best way to do this is to examine the business, its strengths, its weaknesses, its assets, and its value to customers. You must find ways to improve production, cut costs, increase revenue, and resonate better with your customer base. What are you doing right? What are you doing wrong? Are you leaving money on the table by not cross-selling, upselling, or enhancing the customer experience with better service and new or upgraded products? You must find some way to contribute in a more meaningful way to help solidify your stake in the company’s success. I guess that’s a fancy way of saying that you need to participate actively in increasing your company’s bottom line. Be aware that not every idea is going to be great. It’s likely that your management will reject most of your ideas. Don’t allow rejection to discourage you. Brainstorming is how people discover great things, so keep at it. Also, remember that everything works on paper. I’ve had some incredible ideas that worked perfectly well on paper, but two minutes in front of someone else’s eyes yielded a disappointing, “Yeah, but what about X?” Too often my response was, “I hadn’t thought of that. I guess I need to examine this more carefully.” Thomas Edison didn’t create the light bulb on his first try, and the Wright brothers flew more than 1,000 glider test flights before putting engines on their first powered airplane. One of the things you must do is write down your ideas as you think of them. Write each new idea at the top of a page. This way you can expand on your idea, list the pros and cons, and write out how you can implement or deploy your plan as you develop it. Don’t be afraid to list crazy things in your notebook. No one else ever needs to see it. Pres- ent your best ideas to your manager, showing that you have considered the idea from different perspectives. Create a presentation along with your idea to show that you have presentation skills and that you’re really putting some effort into these new ideas, whose goal is to help the company. You and your manager will both appreciate your ideas and analyses, even if they’re rejected. As a real-world example, I had a former co- worker who went through this exact process, made his presentation, and not only gained acceptance for his idea but was promoted to my manager’s manager in the process, where he implemented his plan. I give him props for doing this, although it ultimately led to my leaving the company because of our difference of opinion on our team’s goals and direction. However, the experience taught me something valuable: Present your ideas in an organized, determined manner and see what happens. Sometimes it works out. I hope that if you decide to stay with your current company, you take some of the advice I’ve given to help you remain gainfully employed. You’ll no doubt see layoffs and voluntary attrition in the process, but in the end, sometimes staying the course where you are is the right thing to do. Ken Hess • ADMIN Senior Editor WWW.ADMIN-MAGAZINE.COM ADMIN 70 3

SERVICE Table of Contents ADMIN Network & Security Features Tools Tools Nothing is so true in IT as “Prevention is better than the cure.” We look at Save time and simplify your workday 32 PowerDNS and MariaDB three ways to prepare for battle. with these useful tools for real-world Combining the PowerDNS Authorita- 10 XDR systems administration. tive server daemon with MariaDB’s Extended detection and response multiprimary Galera cluster allows a integrates security functions across 22 Apache Mahout simple yet robust solution for your endpoint devices and networks. But This distributed linear algebra DNS needs. is XDR the only integrated approach framework delivers new tools and to cybersecurity challenges? We methods for performing data analysis, 36 Dogtag investigate the new technology. building machine learning data This certificate manager integrates pipelines, and implementing machine into the FreeIPA open source toolset 16 Defense Against Ransomware learning models in production. to generate SSL/TLS certificates for The possibility of a ransomware attack intranet services and publishes them means it is essential to prepare for 26 Foundries.io on the network. cyberattacks by putting defense A modular system for companies mechanisms and contingency plans in wanting to develop Internet of Things 42 Prometheus Anomaly Detector place. applications for devices. The Prometheus time series database automatically detects, alerts, and 20 Security Through DNS News forecasts anomalous behavior with A holistic approach to designing the Fourier and Prophet models of the network architecture and cybersecurity Find out about the latest ploys and Prometheus Anomaly Detector. uses DNS for cyber defense to detect toys in the world of information attacks at an early stage and fend them technology. 46 Orchestration with Puppet Bolt off before major damage takes place. This free software automates adminis- 8 News trative tasks to speed up the admin’s • Hybrid IT leads to complexity and lack daily work. of visibility, report says • 8 admin tasks to automate Service • Databricks fully open sources Delta 3 Welcome Lake 6 On the DVD • Job changes mean higher salaries for 97 Back Issues Cloud professionals 98 Call for Papers • Serverless architecture lags in adoption 4 ADMIN 70 WWW.ADMIN-MAGAZINE.COM

Table of Contents SERVICE 22 Apache Mahout Distributed 68 Azure AD with Conditional Linear Algebra Framework Access Mahout performs high-volume parallel Trust is good; controls are better. computation on multiple software and Conditional Access sharpens the hardware systems. blurred boundaries that soften models of trust. Containers and Virtualization Nuts and Bolts tm Virtual environments are becoming Timely tutorials on fundamental 9.0 MINIMAL faster, more secure, and easier to set techniques for systems up and use. Check out these tools. administrators. O10-year support life cycle OPHP 8.0 50 Azure Automation 72 AD Sites and Services O Go toolset 1.17.7 A cloud-based service for handling Active Directory domains distributed O Linux kernel 5.14.0-70 automation tasks, managing updates across multiple physical locations with O OpenSSL 3.0.1 for operating systems, and configuring IP subnetting and network configuration O OpenSSH 8.7p1 Azure and non-Azure environments, allows for replication and universal user O SELinux performance with a focus on VM update logins. management and restarting VMs. improvements 78 macOS Shortcuts App O Automatic configuration 56 Jaeger The automation tool of the iPhone and The various components of cloud- iPad moves to macOS Monterey 12 to of security compliance native applications are always help users make their everyday work settings exchanging information, which makes more convenient. O NetworkManager key troubleshooting difficult. The Jaeger files for new profiles tracing framework helps hunt down O GCC 11.2.1 the perpetrators. O Go 1.17.7 Security 84 Performance Health Check Many HPC systems check the state of Use these powerful security tools a node before running an application, to protect your network and keep but not very many check that the intruders in the cold. performance of the node is acceptable before running the job. 62 Teleport Centrally manage logins against various protocols, including SSH, Kubernetes, and databases. Functions such as two- factor authentication are included in the scope of delivery, as is management of your own certificates. 68 Microsoft Cloud Zero Trust 90 Weka See p 6 for details Azure AD with Conditional Access This open source tool applies a makes use of components such as wide variety of analysis methods to device management, risk assessment, data without the need for advanced and user roles to create a new mindset programming skills and without having for zero trust. to change environments. WWW.ADMIN-MAGAZINE.COM ADMIN 70 5

SERVICE On the DVD Rocky Linux 9 (x86_64) On the DVD Rocky Linux is an open source, community-developed, production-ready distribution supported by the Rocky Enterprise Software Foundation (RESF). The enterprise operating system is “100% bug-for-bug compatible with Red Hat Enterprise Linux” [1] and has a 10-year support life cycle with regular updates. Rocky Linux comes with the cloud native Peridot build system, which allows you to extend or reproduce Rocky Linux from scratch – thus avoiding end-of-life issues – and manages infrastructure and secure material like keys and secure boot shims. In version 9, you’ll find: Q Linux kernel 5.14.0-70 Q OpenSSL 3.0.1 Q OpenSSH 8.7p1 Q SELinux performance improvements Q Automatic configuration of security compliance settings Q NetworkManager key files for new profiles Q GCC 11.2.1 Q Go 1.17.7 The Rocky Linux team recommends a fresh install of major versions rather than upgrading older versions. DEFECTIVE DVD? Resources Defective discs will be replaced, email: [email protected] [1] Rocky Linux: [https://rockylinux.org] [2] Build system source code: [https://github.com/rocky-linux/ While this ADMIN magazine disc has been tested and is to the best of our knowledge free of malicious software and defects, ADMIN magazine cannot peridot-releng] be held responsible and is not liable for any disruption, loss, or damage to [3] Networking changes: [https://access.redhat.com/ data and computer systems related to the use of this disc. documentation/en-us/red_hat_enterprise_linux/9/html/9.0_re- lease_notes/new-features#enhancement_networking] [4] Release notes: [https://docs.rockylinux.org/release_notes/9_0] [5] Known issues: [https://access.redhat.com/documentation/ en-us/red_hat_enterprise_linux/9/html/9.0_release_notes/ known-issues] 6 ADMIN 70 WWW.ADMIN-MAGAZINE.COM



NEWS ADMIN News News for Admins Tech News Hybrid IT Leads to Complexity and Lack of Visibility, Report Says Nearly half of technology professionals (49%) say the continued adoption of hybrid IT computing is leading to more complexity for IT management, says Sean Michael Kerner, detailing findings from the recent SolarWinds IT Trends Report 2022 (https://it-trends.solar- winds.com/#/). “The complexity comes in multiple forms,” writes Kerner, “including the continued require- ment to maintain legacy stacks and the fragmen- tation between old and new technologies.” Additionally, 49 percent of survey respon- dents noted that “visibility into their IT opera- © Claudia Soraya, on Unsplash tions for infrastructure and applications has been diminished.” This lack of visibility and monitoring capabilities leads to organizational gaps, says Kerner, in- cluding the ability to effectively conduct anomaly detection and root-cause analysis and to gather metrics from disparate systems. Learn more at Data Center Knowledge (https://www.datacenterknowledge.com/cloud/solar- winds-it-trends-report-reveals-hybrid-it-complexity-challenges). 8 Admin Tasks to Automate Many system administration tasks can be au- tomated to improve your team’s productivity, efficiency, and precision, says William Elcock. For example, manual tasks that admins regularly repeat should be automated to save time and Get the latest reduce human error. IT and HPC news in your inbox This article describes eight top tasks that sys Lead Image © vlastas, 123RF.com Subscribe free to admins should consider automating, including: ADMIN Update and HPC Update • Patching bit.ly/HPC-ADMIN-Update • Password resets © tomatisch zusammen, 123RF.com 8 ADMIN 70 • Disk usage scans Learn more at ServerWatch (https://www. serverwatch.com/guides/system-administrator-tasks-to-automate). WWW.ADMIN-MAGAZINE.COM

ADMIN News NEWS Databricks Fully Open Sources Delta Lake Databricks has announced (https://finance.yahoo.com/ news/databricks-announces-major-contributions-flag- ship-153000807.html) that it will contribute the entirety of its Delta Lake storage framework to the Linux Foundation and open source all Delta Lake (https://delta.io/) APIs as part of the Delta Lake 2.0 release. The Delta Lake framework enables building a “Lakehouse architecture” on top of data lakes (https://databricks.com/dis- cover/data-lakes/introduction). It provides ACID transactions for concurrency control, scalable metadata handling, and unifies streaming and batch data processing. The new 2.0 release of Delta Lake (https://github.com/delta-io/delta/releases/tag/v2.0.0rc1) © Xavier Galle, 123RF.com features improved query performance as well as general improve- ments for writing large scale performance benchmarks. Databricks also released MLflow 2.0,(https://docs.databricks.com/dev-tools/api/latest/ mlflow.html) which includes a new Pipelines feature to accelerate and simplify ML model deploy- ments. The company additionally introduced Spark Connect, which allows Apache Spark to run on any device, and Project Lightspeed, a next-generation Spark streaming engine. Job Changes Mean Higher Salaries for Cloud Professionals O’Reilly’s 2022 Cloud Salary Survey indicates that changing © Helder Almeida, 123RF.com jobs can result in a significant salary increase — of 20 per- cent or more — for cloud professionals, reports FOSSlife. Other results include: • Survey respondents earn an average salary of $182,000. • 20% of respondents reported changing employers within the past year. • 63% of respondents work remotely full time. • 94% work remotely at least one day a week. This year’s survey was limited to U.S. participants and compared salary results by state, education level, age, job title, and certifications earned, among other things. See more results at FOSSlife(https://www.fosslife.org/job- changes-drive-higher-salaries-cloud-professionals). Serverless Architecture Lags in Adoption © bayberry, 123RF.com Containers and microservices are used three times as of- ten as Functions-as-a-Service and serverless architecture, according to recently published research (https://www. digitalocean.com/currents/june-2022) from cloud infra- structure provider DigitalOcean. “The trifecta of containers, container orchestration sys- tems like Kubernetes, and microservices are commonly used in the workplace, but serverless and Functions-as-a- Service lag behind in adoption by organizations,” reports Lawrence Hecht. See more at The New Stack (https://thenewstack.io/ serverless-usage-not-popular-in-workplaces-digitalocean- survey-reports/). WWW.ADMIN-MAGAZINE.COM ADMIN 70 9

FEATURE XDR Extended detection and response in networks, endpoint devices, and the cloud Searching for a Cure Extended detection and response (XDR) integrates security functions tablets), but go beyond that to work- across endpoint devices and networks. But is XDR the only integrated loads in the cloud. In other words, approach to cybersecurity challenges? We investigate the new this definition is genuinely broad, far technology. By Martin Kuppinger beyond the scope of EDR and NDR. XDR collects data from various sys- Information technology (IT) is indis- In this environment, can improved tems and then correlates and provi- Lead Image © Kritiya, 123RF.com pensable for core processes in com- and more powerful integrated solu- sions the data in a structured manner panies that face a tremendous threat tions such as extended detection and for downstream analysis. One key part to their IT systems. Cybersecurity has response (XDR) be understood, and of XDR’s functionality is automatic moved beyond the IT department to what exactly do you need to under- detection of threats, including com- become a central management task. stand these solutions? plex threats that only become visible Laws, regulations, and the associated through an analysis of data across rules of critical infrastructures (CRITIS) Devices and Networks multiple devices and networks. The make it clear how great is this threat detected threats are analyzed, sorted, and the need for suitable countermea- XDR as a term emerged in 2018 and and prioritized so they can then be sures. Manufacturers and service pro- is attributed to software vendor Palo dealt with in a targeted manner. On viders have long since responded with Alto Networks. As the term implies, the basis of this analysis, it is then an almost countless range of products it is about extending existing systems possible to react to possible attacks. and services, from traditional software and detecting, identifying, and re- On the one hand, XDR’s value products such as antimalware to artifi- sponding. The integrated approach promise stems from its integrated ap- cial intelligence (AI)-based systems for is not inherent in this term but is an proach, which is designed to detect identifying security incidents and the important implicit component. XDR even complex threats better by corre- complete operation of security opera- systems are typically offered as soft- lating data from a variety of different tions centers as a service. ware as a service (SaaS), although systems. At the same time, vendors One of the biggest challenges is not this is not a requirement in terms of tout the benefits of SaaS-based inte- the lack of suitable technology, but strategy. grated products that are implemented how to use it correctly and the per- The extension part in XDR specifi- quickly, instead of a multitude of sonnel and knowledge required to do cally refers to endpoint detection and standalone systems that would first so. Even where technology is good response (EDR), as well as network need to be linked together. The basic and powerful, it still has to be used detection and response (NDR). XDR idea behind this process makes sense, properly, and the skills gap (i.e., the now creates approaches that focus on especially if you look at the situation lack of personnel and knowledge) has both endpoints and networks, where in many organizations today, with a long been a central issue, especially endpoints are by no means just cli- large number of IT security products in the complex field of IT security. ent systems (e.g., notebooks, PCs, in use as isolated solutions, generat- ing a great deal of overhead in terms of both licensing and operating costs. 10 ADMIN 70 WWW.ADMIN-MAGAZINE.COM

XDR FEATURE Figure 1: XDR is the combination of a variety of technologies. detection, and response (EPDR) or as an endpoint protection platform Detection and Response detection and three detailed response (EPP). NDR analyzes data from net- phases: isolation, deception or diver- works; leverages threat intelligence XDR is not a technology for all as- sion, and threat elimination. information, including information pects of IT security. As the term XDR focuses on the detect and re- from external entities; and performs suggests, the focus is on the phases spond phases (i.e., respond, isolate, correlations – typically by established of detection and reaction – or in the deceive, and evict). However, for a ho- static and analytical techniques, as more common wordage: response. listic approach to IT security, the other well as machine-learning-based ap- Currently, the two most popular phases (e.g., as described in NIST CSF proaches. The goal is always to iden- frameworks for IT security – National and MITRE D3FEND) need to be in tify deviating and critical patterns and Institute of Standards and Technol- place. In addition to this indispens- to derive concrete indications of pos- ogy (NIST) Cybersecurity Framework able protection, the ability to recover sible threats and tangible suggestions (CSF) and MITRE D3FEND – address quickly also plays a central role, espe- for possible countermeasures. central areas, and the protection side, cially when faced with ransomware. Part of the value proposition of XDR which has long been in the fore- is that it identifies potential threats ground, is supplemented by XDR. Integrated Technology proactively and detects unknown One way to view these technolo- Platform threats (i.e., threats that were not gies is that XDR involves extending previously known or documented) protective measures (e.g., firewalls, Like any popular IT technology, XDR by anomaly analysis, making it more antimalware, and other solutions) to is interpreted quite differently by dif- active than EDR and network traffic include continuous analysis of data to ferent vendors. A basic understanding analysis (NTA). That said, many of detect potential threats and provide exists regarding the combination of today’s EPDR and NDR systems use a targeted response, and NIST CSF network- and end-device-related func- comparable approaches, just without describes the established cycle from tions and the correlation and analy- the integration approach that XDR risk identification, through protection, sis of events across these different uses. EPDR works similarly, but with to detection, response, and recov- domains. In terms of the functions a focus on device usage. EPDR tra- ery. Various established frameworks implemented and of the scope of the ditionally comes from the client area and standards such as ISO 27001 analytic functions, the solutions have but, in the meantime, especially in are referenced (e.g., to identify and some considerable differences. Figure the XDR environment, has developed describe the risk areas and protec- 1 provides an overview of core func- significantly beyond clients. tive measures). MITRE D3FEND is tions and important integrations. Other technologies found in XDR sys- clearly more technical in nature and As already mentioned, the manda- tems include cloud workload protec- primarily focuses on specific technical tory functions in XDR are NDR and tion platforms (CWPPs) for analyzing measures, with the first phase being EDR. EDR products are now typi- and protecting functions delivered hardening the systems, followed by cally offered as endpoint protection, through cloud services, distributed deception platforms (DDPs) for auto- mating the process of creating sitting duck systems to distract attackers, and vulnerability management sys- tems (VMSs) for detecting vulnerabili- ties in the IT infrastructure. XDR also interfaces with user behavior analytics (UBAs) and user and entity behavior analytics (UEBAs) for detect- ing anomalies in user behavior, with unified endpoint management (UEM) for managing and securing endpoints, along with identity and access manage- ment (IAM) for managing users, their authorizations, and, in particular, au- thentication information that is impor- tant in the context of security analysis. Last but not least, of course, is the need to integrate with threat intelli- gence platforms that provide informa- tion on current threats and update it WWW.ADMIN-MAGAZINE.COM ADMIN 70 11

FEATURE XDR Figure 2: The challenge in IT security is to minimize and target unclear events. continuously. XDR systems need to the unknown events that require fur- increasingly common in the NDR adopt this information immediately ther analysis by humans. A good XDR area is integration with operational when analyzing the acquired data to solution must reduce this section to the technology (OT) environments. Cloud respond quickly to new threats. extent possible, while providing good workload protection platforms as an advice as to how to handle the event. XDR building block provide the inter- Integrated XDR One of the essential functions in ana- faces to common cloud environments. lytics is the ability to handle encrypted XDR is a complex, multilayered tech- XDR environments should provide data and either decrypt the data or nology precisely because it integrates a wide range of capabilities and not draw conclusions by reference to the and extends a variety of existing IT just a mere bundling of separate prod- metadata. Additionally, a wide range security technologies (Figure 3). ucts. Because XDR is typically a SaaS of analytical capabilities must be avail- However, as I mentioned earlier, the service, functionality should include able to identify specialized forms of functional differences between solu- unified licensing, subscription, and attack, such as an accumulation of tions are significant, which is what integrated deployment. Centralized unusual DNS requests or an unex- makes a thorough investigation indis- dashboards that visualize the threat pectedly high or low volume of port pensable. landscape are also a mandatory re- scanning operations. The broader the quirement for XDR products. Events analytical capabilities, the more likely XDR, SIEM, and SOAR need to be consolidated across the complex attacks will be detected. various networks and systems. On the other hand, an XDR system’s One question that continually crops A high level of performance in cor- performance is also determined by up in the context of XDR is how it relating and analyzing information to the width and depth of the sensors relates to and interacts with security provide usable information is a cen- (i.e., the components that collect data information and event management tral feature. One of the key challenges on the network or on end devices). (SIEM) and security operations, in IT security is that, out of the huge An important component of XDR now automation, and response (SOAR) number of signals collected, the truly critical threats need to be analyzed to Figure 3: XDR uses a variety of information sources and analytics technologies. find a response. The analysis results of XDR (Figure 2) can be divided into three groups. The events shown in black are clear and known threats. Ideally, the response can be automatic (e.g., by redirecting the event to decoy systems by DDP technologies to render the attacks inef- fective). The events shown in white are clearly identified as non-critical. In individual cases, an automated response might still be needed, for ex- ample, by IT operations management (ITOM). The middle, gray area is the most problematic because it contains 12 ADMIN 70 WWW.ADMIN-MAGAZINE.COM

XDR FEATURE systems. The boundaries between an external vendor who also responds XDR system operations depends on these technologies are fluid. to any security incidents. MDR is the skill set available within the or- First and foremost, SIEM platforms therefore not primarily a matter of ganization. Moreover, the offered ser- collect security-related information technology, but of operations. The vices are never exclusively about the and events from different systems and same applies to SOCaaS, which in- XDR solution. other sources. The value proposition volves offerings in which service The decision must always be made of SIEM also includes continuously providers assume responsibility for within a higher level framework (i.e., analyzing this information to act on setting up and operating security op- with a view to the existing and future it. In this respect, SIEM and XDR erations centers for multiple clients, overall architecture of the IT secu- are closely related, with SIEM typi- typically in a defined interaction with rity solutions with SIEM and SOAR), cally seen as a source of information internal IT security team staff so that which is prerequisite to an organiza- for data that XDR systems process, customer-specific requirements and tion arriving at both manageable and although it can also serve as a target applications can be addressed. affordable solutions that focus on system for information from XDR. Whereas MDR focuses on techni- helping to identify and address critical Ultimately, however, the precise na- cal threats, SOCaaS encompasses a risks to the extent possible. Moreover, ture of the interaction also depends broader range of services, including a holistic concept needs to include the on whether a SIEM solution is pri- SIEM and SOAR system operations phases that go beyond detection and marily used as a plain vanilla data- and security technologies such as response (i.e., identification, protec- base or whether analytical functions next generation firewalls (NGFWs). tion, hardening, and recovery) in case are also deployed to some extent. However, a SOCaaS approach also an attack causes damage. SOAR, on the other hand, mainly fo- specifically includes incident response cuses on operations and the response management (IRM; i.e., incident Conclusions to threats, with SOAR products also preparation and structured security collecting information from a variety incident response), whereas MDR is XDR is an interesting and logical de- of sources. External sources with up- primarily focused on threat analysis to-date information on threats (threat and response at a technical level. velopment in the technology space intelligence) are particularly impor- Again, the boundaries are fluid, as is tant. SOAR systems are typically used so often the case. because it integrates different technol- in combination with XDR and SIEM, Managed security service provid- with an increasing convergence of ers (MSSPs) also have offerings that ogies in a meaningful way to provide SIEM and SOAR emerging. provide additional services, such as Ultimately, before deciding on the vulnerability assessment, application a holistic view of security threats. Be- specific solution portfolio, you will and code security analysis, penetra- need to ascertain which functions are tion testing, IAM operation, and other fore you look into XDR, however, you needed, what the priorities are, and services. which systems are already in place. first need to define an overall concept This requires a portfolio review of Self-Operated XDR the existing IT security products to that includes both the technical ar- ensure that you are not just adding For many organizations, the question one more system, but sensibly inte- is whether they are even capable of chitecture and modular solutions to grating a deliberately limited number effectively and efficiently running an of solutions. XDR, with its integrative XDR environment themselves. In the be deployed and, in particular, the approach, can play a central role in vast majority of cases, the answer this process. What is also important is going to be “No,” because XDR operating concepts. Without such an in this analysis and decision-making requires a high level of skills and up- process is how the technology will be to-date knowledge of security threats overall picture, XDR is just another operated. – even if the application manages to provide concrete, usable threat intel- isolated solution that fails to deliver Security as a Service ligence. Even then, employees need to understand the intelligence and the promised value in terms of IT se- This question brings to the fore the respond appropriately. interplay between XDR, managed In these cases, cooperation with ser- curity improvements. detection and response (MDR), and vice providers proves useful, because security operations centers as a ser- they can draw on expertise. Whether Additionally, XDR’s integrative ap- vice (SOCaaS). MDR describes an this takes the form of an MSSP ap- approach in which XDR environments proach, a SOCaaS offering, or simply proach always involves focusing on and other systems are monitored by MDR for the more technical side of your choice of solution provider (i.e., the risk of dependence on the ven- dor). Interfaces to other products and strategies that enable a change of pro- vider therefore also need to be taken into account from the outset. In any case, organizations need to review the status of their IT security organization and infrastructure regularly, including analyzing if and where technologies such as XDR, MDR, or SOCaaS can help them reduce threats. Q Author Martin Kuppinger is the founder of and Principal Analyst at KuppingerCole Analysts AG. WWW.ADMIN-MAGAZINE.COM ADMIN 70 13





FEATURE Defense Against Ransomware Preparing for cyberattacks The Enemy in My Web The possibility of a ransomware attack means it is essential to prepare for cyberattacks by putting defense mechanisms and contingency plans in place. By Matthias Wübbeling The number of cyberattacks with protocol, and the data on these sys- WannaCry outbreak, many systems ransomware has been rising steadily tems were encrypted. The malware had not yet been updated and were for several years. WannaCry ransom- used a US National Security Agency therefore still vulnerable (Figure 1). ware attacks caused quite a stir in (NSA) exploit named EternalBlue More or less by accident, British se- 2017. Hundreds of thousands of Win- published for propagation by a hacker curity researcher Marcus Hutchins dows systems were infected through group. Although Microsoft released found a way to disable WannaCry. a vulnerability in Microsoft’s SMB a patch to close this gap before the The malware checks for the exis- tence of a special domain before Figure 1: Game over. If you see a message like this on the screen, ransomware – in this case encrypting files. If the domain is not Lead Image © Natalia Lukiyanova, 123RF.com WannaCry – has struck. accessible, WannaCry starts encrypt- ing. After the registration of this do- main in the worldwide DNS system, further propagation was temporarily stopped after just four days. By then, Bitcoin payments equivalent to sev- eral hundred thousand dollars had already been transferred to the at- tackers’ wallet. The encryption of more than 30 servers on the computer network of University Hospital Düsseldorf in the fall of 2020 by a modified WannaCry variant attracted a great deal of attention in the German me- dia. Because IT was unavailable at the hospital, one patient likely died because she could not be admitted and had to be transported to a hos- pital further away. 16 ADMIN 70 WWW.ADMIN-MAGAZINE.COM

Defense Against Ransomware FEATURE Access Vectors groups. After postmortem investiga- Cyber Insurance tions and publications, the hacks In most cases, the goal of ransom- are repeatedly attributed to hacker If you want to protect yourself against the ware is to extort a ransom. Emotet [1] groups in Eastern Europe or Rus- consequences of cyberattacks, you can and EternalBlue [2] are just two of sia. This attribution primarily relies resort to the classic means of insurance and many ways criminals take control of on analyzing the malware and the take out cyber insurance. Depending on the computers and encrypt existing files. perpetrators’ communication with industries in which you operate, cyber insur- The Emotet botnet was dismantled their victims, but small groups of ance can be a beneficial addition to your risk by Europol investigators in early 2021 perpetrators also use ransomware, al- and contingency plan. Insurance companies and the infrastructure was shut down. though they do not develop it them- can also help you take initial steps after an However, this huge success on the selves. Ransomware construction kits attack has occurred and refer proven inci- part of the European authorities only can be found on various forums for a dent analysis partners. Whether or to what alleviated the threat situation for a small sum. extent the consequences of the attack are short while. The first modified Emotet If you look at the public coverage covered and whether the insurance company variants were already in circulation of ransomware incidents, you can’t will pay a ransom depends on your individual by November 2021. help feeling that it is mainly large contract. However, some major insurance In addition to email and security institutions and public bodies that companies announced last year that they vulnerabilities, leaked employee are targeted. However, this image is would no longer pay ransom to criminals. login credentials offer a regular deceptive. Small and medium-sized gateway into corporate networks, enterprises (SMEs) are just as much access corporate networks, which in as illustrated by the Colonial Pipe- victims of ransomware as are private turn means they have found another line case in the US [3]. Attackers individuals. SMEs in particular some- victim for their malware. apparently found valid account times suffer massive damage after cy- credentials for an employee on the berattacks. No official figures indicate Protections Darknet, accessed the company’s in how many cases the ransom was computer systems over a virtual pri- paid; however, the continued high Different measures are required to vate network (VPN), and installed number of attacks is an indication protect yourself successfully against ransomware. As a result of the en- that the method is successful. ransomware and the possible con- crypted files and the ransomware, Although the perpetrators are often sequences of an infection (see the Colonial shut down the entire pipe- not choosy in their victims, they do “Cyber Insurance” box). Technical line system for more than five days. take a targeted approach after a suc- vulnerabilities, such as WannaCry The resulting shortfall in fuel supply cessful raid by analyzing the network and EternalBlue, can be removed led to a sharp rise in fuel prices in infrastructure so they can hijack as by regularly updating all systems in some parts of the US. many systems as possible. More- the company. With regard to auto- Colonial paid the equivalent of over, backup systems are identified mated updates, you first need to ask nearly $5 million in ransom to to make the recovery process more yourself for each system whether the extortionists. However, the complicated. Only after the attackers a potential failure because of an extortionists did not simply rely have grabbed the sensitive data does error during the update process on encrypting Colonial’s files. In- the encryption process begin, with outweighs an existing security vul- stead, they copied almost 100GB the ransom demand following on its nerability. You will want to enable of files from the Colonial systems heels. automatic updates where possible, beforehand. The blackmailers ad- Although private computers might even at the risk of failure because of ditionally threatened to publish this not seem particularly worthwhile tar- a failed update. The longer it takes information if Colonial did not pay gets at first glance, the consequences to review and release an update, the the ransom. This practice of copying can be explosive – even if the owner more time attackers have to access files before encrypting them opens does not pay a ransom – because enterprise systems. another attack vector. Affected the attackers also collect passwords You will want to use centralized anti- companies will still pay even if the from the usual password safes of the virus and application layer gateways data can be easily restored from a browser or email program. Although to scan email attachments before they backup to prevent the publication of these passwords are often secured by are delivered to employee accounts. internal data and possible company a master password, the master can be At least the malware variants cre- secrets. sniffed. In many cases, access creden- ated by construction kits can often be tials for other computer systems are detected in this way, although they Perpetrator and Victim also preserved in these records, which often do not work against the individ- is how criminals repeatedly find login ual variants of the larger groups. The perpetrators of large ransomware data for remote maintenance access. To prevent retroactive loading of mal- incidents are mostly well-organized The perpetrators then use the data to ware after an infection, you can redi- rect your employees by web proxies, WWW.ADMIN-MAGAZINE.COM ADMIN 70 17

FEATURE Defense Against Ransomware prohibit binary downloads or enable signed macros and prevent unsigned new file for each change, but the user them separately, and initially block macros from running. cannot delete or encrypt the files. all other requests to pass by the proxy The principle of least privileges in the packet filter. Security product Restricting Access means, above all, that you need to es- providers offer lists of IP addresses tablish processes that regularly check and domains that you need to filter Valid login credentials in the hands the existing privileges, especially for this purpose. These measures of criminals, in addition to technical when employees change departments often cannot be implemented with- vulnerabilities, are a major problem or collaborate across departments on out restricting employees, and they for the security of your comput- projects. The phenomenon is com- sometimes interfere with daily work. ers and services. On the Darknet, mon, for example, among interns Therefore, many companies do with- hackers can obtain extensive col- who pass through different depart- out them. However, it makes sense lections of identity data and login ments in the course of their intern- to work with other staff to see which information. Because users tend to ship. Once granted, privileges are measures you can implement. You use the same passwords for different often not revoked, but new ones are can also take into account the times services, you might also be at risk if regularly added. At the end of an in- when no one usually works. When other services fall victim to hacking. ternship, the intern then has access your office is closed, you can imple- In fact, more than two-thirds of us- to a user account with many security- ment and monitor far stricter rules, ers continue to use previously leaked related access options. which are then relaxed again during login data for more than a year. The normal business hours. US National Institute of Standards Preventing the Spread Even though a company’s users are and Technology (NIST) and the Ger- always portrayed as the highest risk man Federal Office for Information If attackers do gain access to comput- vulnerability, they are really your last Security (BSI) IT basic protection ers in your corporate network, despite line of defense and can prevent an in- compendium (IT-Grundschutz-Kom- all protective measures, this does fection, unlike the industrial security pendium) point out the dangers and not necessarily mean that they will products such as antivirus programs, recommend regular checking of user ultimately be successful with their application layer gateways, and spe- accounts and passwords. attack. Try to mitigate the damage in cial firewalls. Different service providers offer iden- these cases. To prevent the spread, Attackers use spearphishing tech- tity leak checkers. The free American it makes sense to isolate different niques and, as in the case of Emotet, service Have I Been Pwned (HIBP) [4] departments and different teams in existing communications to trick us- is probably the best known provider the same department from each other ers into running the malware. Seeing of leak information. By entering an in terms of network technology and through these perfidious techniques email address, you will receive infor- locate them on their own subnets. is difficult even for well-educated and mation about whether it was part of Between these subnets, you need to trained employees. Educating your a data leak. Even if it is your com- have a firewall that regulates interdis- users needs to be an integral part pany email address, the use of HIBP ciplinary network traffic, limited only of your overall IT security strategy. is questionable for data protection to what is necessary. Targeted training (including an active reasons in some countries and should The faster you react, the greater error culture and an option to report be discussed with the corporate legal your chance of averting a major loss. conspicuous activities) sensitizes your department. Moreover, HIBP does Comprehensive monitoring of your users so that they do not involuntarily not give you direct access to the af- resources identify and isolate affected help the attackers. fected password to this account, so systems quickly. You might want to Additionally, you can provide technical you cannot check it directly against isolate an entire team or department support to your employees and estab- your systems. However, specialized together. In this way, you can up- lish email signing in your organization service providers on the market also hold the ability to work and protect to make it at least a little more difficult implement General Data Protection the other organizational units in the to create credible email. If all the email Regulation (GDPR)-compliant checks meantime. Of course, you also need in your organization is signed, those of login data. to go through this process regularly. messages that aren’t will stand out. When assigning user rights, you need Often, only a few firewall rules are re- The fewer exceptions, the more reli- to consider the possibility of stolen quired. Depending on your infrastruc- ably your employees can detect fake login data and give employees only ture setup, you can also automati- email. However, if a user opens a mali- the access rights they absolutely need cally isolate affected computers in a cious attachment, you need to protect for their normal workday, especially separate virtual local area network the system with active group policies when accessing servers and shared (VLAN). An attacker then still has ac- that prohibit the execution of macros files. For example, if you grant users cess to a system but cannot infect any in these files. If users need macros read-only rights for existing files on other computers from there. If you log for their daily work, then at least use a server, the users need to upload a internal network connections on the 18 ADMIN 70 WWW.ADMIN-MAGAZINE.COM

Defense Against Ransomware FEATURE routers between your departments, ransomware incident, you need to process of working out your risk you can even determine afterward respond adequately. Ideally, you and contingency plans, at least try whether propagation – also known as will have drawn up risk and contin- to recover what can be salvaged, lateral movement – has taken place. gency plans in advance and defined including consideration of a ransom Although attackers might have responsibilities. The plans include payment. However, you should ar- penetrated your corporate network information about the criticality of range this in collaboration with the through a vulnerability, it doesn’t individual systems and specify the authorities you notified after the mean they will find the same vulner- extent to which you need to shut incident. Set up a crisis team with ability on other systems. Attackers down other systems. A targeted and all the people you can identify as therefore use different tools, including planned shutdown can protect your relevant in a timely manner and dis- the remote desktop protocol (RDP) company against existential damage, cuss the necessary measures. supplied by Microsoft. Especially in even if collateral damage has to be these times of home office and VPN accepted in the process. The opera- Conclusions connections from home to the corpo- tors of Colonial Pipeline responded rate network, remote desktop connec- in an exemplary manner and specifi- Ransomware is a big threat to busi- tions are enjoying great popularity. In cally removed the system from the most cases, access is quickly granted network. nesses, public institutions, and by Active Directory. Again, you need You need to inform contacts in the to monitor connections established affected departments in good time individuals. In recent years, the with the directory service at central and make sure that backup systems locations and automate your response go online in a way that reflects the consequences of ransomware at- to undesirable connection attempts to established criticality. If you are the extent possible. legally required to report cyber inci- tacks have grown in scale. In this dents, you should have appropriate Protecting Backups forms pre-filled and make a report article, I looked at the various at- in a timely manner. This proce- Creating backups is one of the admin- dure will help you avoid penalties tack vectors and manifestations of istrator’s standard tasks. However, because you left steps out. Keep you should not only handle and the affected systems as-is for later ransomware from actual incidents monitor how backups are created, forensic analysis. You can handle but also how existing backups are this step yourself if your company is and discussed the risk that exists protected and accessed. At best, you large enough and you have appro- have no access to the backup system. priate skills in your IT department; and how contingency plans can Instead, the backup system needs otherwise, commission an external access to the individual services it service provider to perform the help you restore operations when is supposed to back up. If the us- analysis. The main goal is to iden- ers of your systems cannot access tify the vulnerability – one hopes responding to attacks. Q the backup themselves, it cannot be you have been able to restore your encrypted by ransomware launched data from backup. Gradually rebuild Info from a normal user account. your infrastructure once you have [1] Emotet: To support easy recovery of files for eliminated the vulnerability. While normal use, you will want to establish you’re at it, don’t forget to set up [https://en.wikipedia.org/wiki/Emotet] different backup systems: one that new backup systems to cushion the [2] EternalBlue: [https://en.wikipedia.org/ your users can manage themselves, effect of a new attack. and one that can only be accessed In the best of all worlds, you will also wiki/EternalBlue] in extreme emergencies and only by have an internal contingency plan for [3] Ransomware attack on Colonial pipeline: a few administrators. Although this each department that will inform sup- action does not protect you against pliers, partners, or customers in the [https://www.bloomberg.com/news/ an attacker demanding a ransom to respective areas. If you are a supplier articles/2021-06-04/hackers-breached- protect your sniffed trade secrets, you yourself, you need to notify depen- colonial-pipeline-using-compromised- can quickly resume operations after a dent companies in the supply chain password] ransomware incident. in a timely way and inform your own [4] Have I Been Pwned: suppliers in these times of zero-stock [https://haveibeenpwned.com] Regulating Processes supply chains and just-in-time pro- duction. The Author If the cat is out of the bag and your If you were caught off guard by Dr. Matthias Wübbeling is an IT security en- systems have been affected by a the attack while you are still in the thusiast, scientist, author, consultant, and speaker. As a Lecturer at the University of Bonn in Germany and Researcher at Fraunhofer FKIE, he works on projects in network security, IT security awareness, and protection against account takeover and identity theft. He is the CEO of the university spin-off Identeco, which keeps a leaked identity database to protect employee and customer accounts against iden- tity fraud. As a practitioner, he supports the German Informatics Society (GI), administrat- ing computer systems and service back ends. He has published more than 100 articles on IT security and administration. WWW.ADMIN-MAGAZINE.COM ADMIN 70 19

FEATURE Security Through DNS Employing DNS in network security Revealing Traces A holistic approach to designing network architecture and cybersecurity uses DNS for cyber defense to detect attacks at an early stage and fend them off before major damage takes place. By Steffen Eid The corporate network has long alerts among themselves, overwhelm- According to a Ponemon study Photo by Abbas Tehrani on Unsplash ceased to be a single perimeter with ing what are already overburdened sponsored by IBM [1], it still takes branch offices connected to the out- security teams. However, the tool for more than 280 days on average for a side world by the Internet. In the achieving a unified, comprehensive security breach to be detected, but growing network jungle, however, view of your network already exists – containing a breach in under 200 an overall perspective is often dif- the Domain Name System (DNS). Af- days would save $1 million in costs. ficult to maintain, which is why ter all, as the hub of communications dividing the network into individual on the Internet, DNS can be the heart Vulnerable Without silos to give it structure seems of integrated network management Safeguards tempting at first glance. This ap- and security. proach would definitely be wrong, Without DNS, any activity on the because thinking in silos causes More Is Not Always Better web would be messy because DNS problems. Most important is the of- converts the input from URLs into ten missing ability to communicate In IT departments, when workflows the significantly more difficult to re- between isolated solutions because are not fully covered by just one member IP addresses, helping users a wide variety of security tools are security tool, communication inter- access the desired websites. As con- implemented in the silos – and usu- faces need to be kept as up-to-date venient as DNS is for users, it can ally more than one. as possible at all times, and employ- also be dangerous if it is not secured Next-generation firewalls, web gate- ees need to be constantly trained properly, because attackers use DNS ways, email security, endpoint secu- in the use of the many tools. These to communicate with their targets rity – the security solutions in the in- resources could be put to better use and for data exfiltration. Whether dividual sectors are often piled up on elsewhere. This problem is even the attack is meant to steal confi- top of one another. The unintended more pronounced in large enter- dential company data (exfiltration), consequence of this strategy is that prises, which can be geographically infiltrate with malware in small data communication between the indi- widespread and might be working packets (infiltration), or create sepa- vidual systems is poor, and often even on restructuring such as mobile use, rate communication tunnels to make incorrect. For example, if interfaces a multicloud rollout, or software- transferring data even more conve- are not configured correctly, the secu- as-a-service (SaaS) and software- nient, hackers use DNS as an access rity tools can trigger false or duplicate defined (SD)-WAN implementations. vector into networks. 20 ADMIN 70 WWW.ADMIN-MAGAZINE.COM

Security Through DNS FEATURE To understand how DNS can help traffic, or communicate with the the hacker accessed and how they you comprehensively secure your attacker in some other way. Con- moved around the network. In this own networks, you need to look versely, DNS contains all the data way, the attack can be reconstructed back to the early 2000s when DNS you need to detect an attack. De- and retraced holistically. security tended to be a minor con- fenders who keep an eye on their cern. At the time, the Berkeley In- DNS at all times can take advantage Automated Counter to ternet name domain (BIND) servers of this fact, quickly detect unex- Hackers – still an important standard in DNS pected atypical communications, today – had only two security fea- and take countermeasures. Without The most advanced companies and tures: They did not accept responses question, this task is mammoth. Ar- vendors feed DNS telemetry data – from IP addresses they had not que- tificial intelligence helps keep track also known as passive DNS – into ried (also known as Mars responses), and automatically filter out harmful data stores and then have the data and they inserted a random 16-bit communication requests. analyzed by machine learning al- number into outgoing requests and The potential of DNS as a security gorithms. Sophisticated algorithms checked that the number came tool in its own right has only re- can detect various types of malicious back in the responses. Only later cently been recognized. The advent activity in passive DNS data, includ- did analysts discover that this test of response policy zones (RPZs) in ing, for example, requests sent by a number was not really random. Little 2008, meant that DNS servers could domain generation algorithm (DGA), wonder, then, that DNS servers have be leveraged to issue “benevolent which is code that automatically cre- long been a worthwhile target for at- lies” when they received an informa- ates a list of domains used by mal- tacks, such as the Li0n worm, which tion request whose response could ware clients to communicate with exploited a vulnerability in BIND. be damaging to the querying entity. a number of command-and-control Moreover, DNS servers of all kinds At least as important was the abil- (C&C) sites. are often used as amplifiers in dis- ity to detect when a DNS server was These domains serve as a meeting tributed denial of service (DDoS) at- queried for data known to be cor- point for malware- and hacker-con- tacks and are still the target of such rupted. Since then, companies have trolled servers that communicate se- attacks to this day. appeared that prepare DNS threat cretly over a backhaul network. Once That said, over time, the security of data in the form of RPZs and offer one of the DGA domains is detected DNS servers and DNS itself has im- their customers this data commer- and blocked by IT security, the mal- proved. BIND has been optimized cially. Organizations can incorporate ware client and C&C server move to to support access control lists for a variety of RPZ feeds into their DNS the next domain on the list to bypass almost everything: queries, recur- infrastructure and enable their DNS the defenses. For example, the de- sive queries, zone transfers, and servers to protect users and systems fense algorithm can detect patterns in dynamic updates. The DNS commu- against known malware propagation the newly created domain names and nity started to operate DNS servers sites, command-and-control infra- directly identify them as threats. in chrooted environments according structures, and much more. to the principle of least privilege. RPZs are also helpful in centrally Conclusions Additionally, transaction signatures monitoring network health, such (TSIGs) and DNS security enhance- as detecting infections and security DNS is an indispensable part of any ments (DNSSEC) were introduced breaches across the board. A lap- to further protect DNS. Even if DNS top that sends a query to a domain modern security toolkit, playing both itself is not attacked, though, it name that is clearly used by a cer- remains the communication high- tain type of malware is almost cer- an active and a supporting role in way that hackers still use for their tainly infected with that malware. attacks. Armed with this knowledge, impor- securing networks and tracking mali- tant measures can be taken quickly DNS as the First Line and efficiently without the need cious activity. Moreover, DNS is a of Defense for one of the many other security tools to sound the alarm first. The central tool already in place connect- Cyberattacks are as varied as the benefits of central DNS as a security attack vectors available to hack- layer go even further: Organiza- ing all departments, which can fa- ers, but almost all of them have tions that archive all of their DNS one thing in common: They depend query logs have an important tool cilitate the paradigm shift away from on DNS for almost all communica- at hand in the event of an infection. tion on the network. For example, Even if the attacker is not imme- silos and toward a holistic integrative more than 90 percent of malware diately detected, these logs can be uses DNS to exfiltrate data, redirect used to trace which other systems approach. Q Info [1] Cost of a Data Breach Report 2021: [https://www.ibm.com/security/data-breach] Author Steffen Eid is a Manager for Solution Architects in Central Europe at Infoblox. WWW.ADMIN-MAGAZINE.COM ADMIN 70 21

TOOLS Apache Mahout Distributed Linear Algebra with Apache Mahout Matrix Math The Apache Mahout distributed linear algebra framework delivers new tools and methods for performing data analysis, building machine learning data pipelines, and implementing machine learning models in production. By Andrew Musselman and Trevor Grant The ideal scenarios for using with math notation, avoiding the usual to write, and later read, complex Lead Image © Peter Galbraith, Fotolia.com Apache Mahout are in teams with “syntax bloat” that most machine mathematical formulas. Mahout has a the flexibility to adapt as their needs learning libraries require. rich history and was one of the (if not change over time. Mahout can eas- the) original machine learning librar- ily swap back-end compute engines Apache Mahout – What Is It? ies for big data. Originally designed (e.g., batch or micro-batch systems to aid in machine learning tasks on such as Apache Spark) or streaming Apache Mahout is a library designed data in Apache Hadoop clusters, it systems (e.g., Apache Flink). Ad- to make composing and maintaining underwent a major refactoring around ditionally, Mahout is able to perform distributed linear algebra algorithms 2014-2015 that resulted in its current compute on multiple software and easy. First, it creates an abstraction form. One challenge this restructuring hardware systems, ranging from the layer on the underlying engine (the creates is the abundance of informa- Java Virtual Machine (JVM), to what- open source version of Mahout uses tion in circulation that refers to the ever multicore CPU is available, to Apache Spark as an engine), and the “old” Hadoop-based Mahout. on-board GPU for high-volume paral- abstraction layer implements basic Machine learning sometimes refers lel computation. linear algebra functions on datasets to a set of techniques for numeri- Typical users of Mahout are advanced in the engine (e.g., by defining a dis- cally solving problems that are too data engineers and data scientists who tributed matrix, matrix multiplication, large for standard statistical ap- have experience writing transforma- multiplication with self transposed, proaches. Mahout gives you tools tions and models in other languages and other functions). Second, it uses to solve those problems with tested but who are looking for an approach Samsara DSL in Scala, which allows statistical approaches. Mahout can that does not require they rewrite their users to define algorithms with an R- be applied to any “big data” plat- work, depending on the system they like syntax that makes it much easier form (e.g., any platform on which use from month to month or year to year. Moreover, mathematics-oriented Figure 1: A conceptual illustration of the way Mahout approaches matrix multiplication of software engineers will enjoy the matrix A with its own transform. simplified Samsara domain-specific language (DSL), which provides a stripped-down syntax that feels natu- ral to people accustomed to writing 22 ADMIN 70 WWW.ADMIN-MAGAZINE.COM

Apache Mahout TOOLS users are running distributed data- this by introducing sets) and can pay dividends if your an engine abstraction company has critical algorithms of layer, in which some- concern for refactoring into machine one who is an expert learning approaches. in the underlying en- gine will implement Distributed Linear Algebra performant ways to do basic operations Linear algebra, otherwise known as like multiplying two matrix math, is a rich and established matrices, multiply- field of theoretical and applied math- ing a matrix times ematics that has found applications itself transposed, and across multiple spheres of computer other operations. science and software engineering, Figure 2 shows the including visual simulations, audio application layers at analysis, and predictive analytics. the top, with mul- In some cases it can be arithmetic- tiple engines trans- heavy, with methods built up in parently managing iterative or bulk activity, and at scale the distribution Figure 2: Architectural diagram of an engine abstraction layer. these computations val G = B %*% B.t - C - C.t + (xi dot xi) * (s_q cross s_q) Listing 1: Samsara Syntax Examples can become either overly cumbersome // Dense vectors: a -= b or complicated to val denseVec1: Vector = (1.0, 1.1, 1.2) a += 5.0 program on dis- val denseVec2 = dvec(1, 0, 1, 1, 1, 2) a -= 5.0 tributed systems. a *= b Fortunately, Apache a *= 5 Mahout abstracts // Sparse vectors: // Dot product: away many of the val sparseVec1: Vector = (5 -> 1.0) :: (10 -> 2.0) :: Nil a dot b complexities and val sparseVec1 = svec((5 -> 1.0) :: (10 -> 2.0) :: Nil) // Cross product: pitfalls of distributed a cross b linear algebra, leav- // Dense matrices: ing a tidy library val A = dense((1, 2, 3), (3, 4, 5)) // Matrix multiply: for working with a %*% b distributed linear // Sparse matrices: algebra as though it val A = sparse( // Optimized right and left multiply with a diagonal matrix: were simply normal diag(5, 5) :%*% b linear algebra. For (1, 3) :: Nil, A %*%: diag(5, 5) example, in Figure 1, (0, 2) :: (1, 2.5) :: Nil the original matrix ) // Second norm, of a vector or matrix: A and its transpose a.norm AT are sliced into // Diagonal matrix with constant diagonal elements: diag(3.5, 10) // Transpose: independent and cor- // Diagonal matrix with main diagonal backed by a vector: val Mt = M.t responding rows and diagv((1, 2, 3, 4, 5)) // Cholesky decomposition columns, which then val ch = chol(M) can be sent across to // Identity matrix: a compute engine to eye(10) // SVD perform small chunks val (U, V, s) = svd(M) of arithmetic. The // Plus/minus: results are then com- a+b // In-core SSVD piled together for a a-b val (U, V, s) = ssvd(A, k = 50, p = 15, q = 1) single output. a + 5.0 In mathematics, com- a - 5.0 // EigenDecomposition plex procedures are // Hadamard (elementwise) product: val (V, d) = eigen(M) a*b normally the product a * 0.5 // QR decomposition of many simpler val (Q, R) = qr(M) procedures. Mahout // Operations with assignment: takes advantage of a += b WWW.ADMIN-MAGAZINE.COM ADMIN 70 23

TOOLS Apache Mahout business-critical algorithms and meth- ods from an old system to a new one. The Mahout project lived through the migration from Hadoop to Spark and incorporated the lessons learned into its very fabric, making it simple to port algorithms from any arbitrary platform to any other (Figures 3 and 4). Note that the code after the import statements requires no changes. Therefore, a team that uses one back- end engine is able to migrate code onto another engine, without having to change the code performing the mathematical operations and avoiding the more error-prone part of porting code from one platform to another. Figure 3: Matrix math with the Apache Flink engine. Mahout Use Cases of computation for the user. The Engine Agnosticism Mahout is for organizations who have advantage of this approach is that (and Why It Matters) statistical methods to run on distrib- the end user, the person implement- uted datasets but want to minimize ing the algorithms, doesn’t have to Engine agnosticism is important to their exposure to the technical debt know all of the peculiarities of a many organizations, many of whom that arises from writing algorithms specific engine. In fact, the person don’t even realize it. In just the against a specific engine that may or writing the algorithms doesn’t even 2000s, clusters have moved from may not have a successful future. Ma- need to know what the underlying Hadoop to Spark, and from Spark to hout allows organizations to switch engine is. Kubernetes. Teams and organizations their systems of record while having a often find themselves in an uncom- minimal effect on their data outputs. Samsara: The Scala DSL fortable position of being pinned to Mahout also lets users add linear outmoded technology, rather than algebra concepts to data stores that The end user interfaces with Sam- bringing in costly consultants to port have either weak or nonexistent sara Scala DSL, which makes it implementations for linear algebra much more pleasant and natural for mathematicians and the writers of algorithms to implement mathematics with the help of an interesting Scala feature that allows users to change syntax and language rules for a par- ticular use case. For example, the computation below, which is used in Mahout’s distributed stochas- tic principal component analysis (dsPCA), when written with math- ematics notation here, is written with Samsara as shown in Figure 4: Matrix math with the Apache Spark engine. the first line of Listing 1. The lines that follow are more examples of typi- cal statements exercising Samsara syntax. You can find more informa- tion about Samsara [1] online. 24 ADMIN 70 WWW.ADMIN-MAGAZINE.COM

Apache Mahout TOOLS concepts, such as Apache Spark. researchers and practitioners have the project as a user or contributor, (Spark’s linear algebra works fine built recommenders, similarity en- in single-node deployments but has gines, and other predictive models at subscribe to [email protected] issues scaling to larger distributed scale with the use of its tools. datasets.) A paper written by T. Grant [2] il- and [email protected] mail- lustrates another Mahout use case. Mahout in the Wild During the outset of the COVID-19 ing lists (instructions online [5]). For pandemic, CT scans were shown to A major used car marketplace in be as good as, or in some cases supe- some alternative methods for install- North America used the Mahout co- rior to, RT-PCR tests. A major issue debase when creating their car rec- however was the high dose of radia- ing and using the software, including ommendation system. This recom- tion they delivered. With Mahout, mendation engine is based on Ma- Grant showed how “noisier,” “low- a prebuilt Docker image, see the slide hout’s Correlated Cross-Occurrence dose” CT scans could be quickly and (CCO) analysis. The CCO algorithm easily denoised, with about five lines presentation at SlideShare [6]. Q is very similar to the more popular of Mahout code. co-occurence (CO) algorithm, but it Info also incorporates other attributes of Getting Started with Mahout [1] Samsara: [https://mahout.apache.org/ the user into its recommendations; in more technical parlance, it is Mahout can be added to your project docs/latest/mahout-samsara/faq.html] multimodal. by adding it as an Apache Maven [2] “Denoising COVID-19 computed tomog- Mahout has been used in many situ- dependency, by running a prebuilt ations where customer privacy and Docker image, or by downloading a bi- raphy scans with scalable open source intellectual property concerns keep nary [3] or a source build [4] from the software,” by Trevor Grant, them from being published, but many project website. To get involved with [https://nobleresearch.org/Content/PDF/12/ 2399-8172.2020-6/2399-8172.2020-6.pdf] [3] Binary image: [https://mahout.apache.org/ general/downloads] [4] Source build: [https://mahout.apache.org/] [5] Mailing lists: [https://mahout.apache.org/ general/mailing-lists] [6] SlideShare: [https://www.slideshare.net/ AndrewMusselman/apache-mahout-on- zeppelinpptx]

TOOLS Foundries.io Foundries.io IoT development platform Conundrum Solver Foundries.io is a modular system for companies wanting to develop are powerless, because most IoT de- Internet of Things applications for devices. By Martin Loschwitz vices don’t even provide for updates over the network to patch security If you want to see die-hard system includes a software stack, a standard- holes that have become public. Photo by Juan Rumimpunu on Unsplash administrators go mad, all you need ized development environment, and a The service engineers who replace to do is bring up the topic of the In- pricing model that avoids per device ternet of Things (IoT). Virtually no charges. Figure 1: Today, even cars have an Internet other a topic will unite Linux admins connection and communicate over a in such a unanimous opinion. Stories Importance of Security smartphone, making them part of the offer examples of IoT applications Internet of Things. that have gone wildly wrong. Many That said, the mistrust of IoT is not of these are urban legends and never entirely unjustified. Undoubtedly, a actually happened, but they fit beauti- washing machine with a WiFi module fully into the narrative of technology for remote app control that can be hi- that doesn’t really help anyone, opens jacked by attackers on the web is not security holes, and, in the eyes of a good thing. Unfortunately, precisely many, simply shouldn’t exist. this kind of attack has been exten- However, valid IT approaches do ex- sively documented in the operations ist, and IoT devices perform well ev- security (OpSec) scene. Often the ery day. The fact that the nation’s dis- companies that regularly make the counters regularly offer WiFi-capable biggest security mistakes are those cameras with woeful security mea- that have little or nothing to do with sures should not obscure the fact that software. in many places around the world, Just because you build good wash- networked WiFi cameras with good ing machines does not automatically security can protect people. In other make you an innovative IT company. words, IoT applications often are Companies looking to give their not as useless as some people would devices a modern and smart touch make them out to be (Figure 1). regularly source modules externally This is where Foundries.io steps into and install them more or less with the breach by offering Foundries no inkling of what might go wrong. Factory as a complete solution for When the appliance reaches the companies that want IoT functionality customer’s location and turns out to but do not want to deploy the entire have an intrinsic security problem, infrastructure in-house. The package both the customer and manufacturer 26 ADMIN 70 WWW.ADMIN-MAGAZINE.COM

Foundries.io TOOLS relays and bearings in washing ma- ejected the toast after the time set by all the relevant components, which chines are highly unlikely qualified to the user. This would be a fairly trivial is exactly where the Foundries stack help customers install new firmware. implementation. (Figure 3) comes in. Often, manufacturers looking to jump For most IoT developers, though, The innermost core is a Linux ker- on the IoT bandwagon to reach state- this basic functionality doesn’t go far nel that includes support for a wide of-the-art status lack a plan and have enough. Ideally, you would want the range of popular ARM boards for no idea how IoT can be implemented toaster to stop the toasting process IoT deployment. The kernel is en- in a meaningful way. automatically by the level of brown- riched with drivers for chips that are ing set in an app, without having to typically used in the IoT environ- Foundries.io specify a time. This scenario requires ment, for example, (W)LAN devices. significantly more technology, such as The project’s website provides a In essence, the Foundries stack is a a sensor that can measure the degree list of boards [1] that can be used framework into which external de- of browning. Another function allows directly with the software stack pro- velopers can build functionality for the built-in computer to interrupt the vided by the Foundries project. a device while having access in the toasting process and eject the toast as Although this solution does not sound background to the centralized skill set soon as it is ready. Also, the toaster like much in theory, it is, in practice, of a large provider, including the cor- needs a network connection. Mod- a massive boost for companies look- responding logistics. ern devices rely exclusively on WiFi, ing to get started with IoT devices. If you use the Foundries stack, the which requires a matching chip and Thanks to the preparatory work by marketing people promise, you no antenna. Foundries.io, a basic system is avail- longer have to develop the major share The hardware bill of materials leads to able within minutes on which further of an IoT stack for any device your- the conclusion that the software used development can be built, provided self; rather, you can rely on a modular in a device of this type must be capa- system that already contains the most ble of far more than just a few simple Figure 2: Every IoT device (e.g., the important functions. I took an in- network commands. The task is to smart lights by Philips in this example) depth look at Foundries.io, so I could read and interpret values from sen- is basically a small computer with a explain how the solution works, what sors. The operating system – because system-on-a-chip board that requires an its advantages and disadvantages are, even the toaster needs one – needs operating system. Foundries.io offers the and whether the product gives rise to drivers for the sensors and compo- Factory development environment to help hope of more secure and capable IoT nents, and you need some kind of manufacturers achieve results quickly. applications. software in userland that links the features of these components in a What IoT Needs meaningful way. If you don’t think the toaster example What do manufacturers need to es- is practical enough at this point, tablish IoT-enabled devices success- imagine a similar scenario with a fully on the market? I have already smart washing machine or refrigera- touched on a few factors, including tor. The combination of a computer the inability of a company producing board with networking capability and household appliances to develop IoT a variety of sensors are found in al- software in-house and their reliance most any scenario. on off-the-peg components available on the market. Unfortunately, a soft- Stack and Operating System ware stack alone does not make an IoT application; the hardware (Figure To get back to the Foundries stack, 2) also plays an important role. it should first be noted that the A toaster that is supposed to notify product exclusively relates to the the owner over WiFi that the toast is software part of an IoT application. ready is a good way of proving that Foundries.io doesn’t build the hard- IoT applications require more hard- ware, but the manufacturer does main- ware than you might at first think. A tain partnerships with contractors with mandatory requirement is some kind operations in the IoT market. of CPU, and it needs a sensor that IoT devices are almost always em- can detect that the toast is finished. bedded. Accordingly, devices with In the simplest case, the sensor might an ARM system on a chip (SoC) are simply detect that the toaster has widespread; the entire computer automatically switched itself off and comprises a small circuit board with WWW.ADMIN-MAGAZINE.COM ADMIN 70 27

TOOLS Foundries.io Figure 3: Foundries.io provides enterprises with a development platform for IoT applica- devices in the first place. What has tions that has a Linux kernel with a container platform and microservices at its core. been an established standard on © Foundries.io desktop and server systems in the form of the Trusted Platform Mod- that a suitable SoC board is available. Security ule (TPM) for a long time also exists Without Foundries.io, just putting for embedded systems. But most together a suitable Linux distribution For obvious reasons, the issue of providers do not make use of these for embedded devices would take a security plays a major role for IoT options. medium-sized team months. devices. On one hand, these devices Foundries.io approaches the prob- are not shielded from the outside lem differently: It fully supports the More Than Linux world as much as you might as- security features of any hardware sume. Recall once again the exam- on which it can run. From the boot- Linux, by the way, is not the only ple of surveillance cameras: Because loader to individual drivers and operating system with which the they support the Universal Plug and programs, a chain of trust can be Foundries developers planned to Play (UPnP) standard and many created that prevents the execution work. During its startup phase, they routers for domestic use are config- of arbitrary code, even if an attacker dropped quite a few hints in the ured to pass automatically through is working as root on the system. documentation and online that the ports released by UPnP to the out- The Foundries stack scores bonus company had its sights set on an side world, the corresponding cam- points because it can also natively embedded distribution based on the eras suddenly become accessible on use the cryptographic functions of Zephyr real-time system. Zephyr, like the web. many ARM and Intel chips on the the Linux kernel, is under the aegis of On the other hand, many owners market to enable efficient encryp- the Linux Foundation and specializes would not even notice an attack on tion. Therefore, developers can use in real-time computing. smart home devices, as long as the encrypted connections instead of In the meantime, however, the refer- basic functions of the device are not plain text, further contributing to ences to Zephyr have disappeared affected. If an attacker were able to device security. from the vendor’s documentation and take control of the IoT toaster de- website, and the Zephyr-based dis- scribed above and a few thousand Updates tribution is probably no longer main- more devices with the same vulner- tained. However, it would only have ability to execute arbitrary code, it Updates for IoT devices are a com- appealed to a relatively small group would even be possible to imagine a plicated affair. In practice, they can of users anyway, because real-time botnet populated just by toasters. only occur “over the air” (OTA; i.e., computing is only likely to play a Several approaches are available to in a way that does not entail the pro- minor role given the typical use cases prevent this kind of attack. For ex- vider having physical access to the in the IoT environment. The develop- ample, known vulnerabilities could device after delivery). The distribu- ers are also aware that an operating be repaired by patching. Another tion model for updates accordingly system kernel is not the same as a approach would be not to allow the provides that they be made available functional IoT framework. execution of arbitrary code on the on central servers so that clients can download them autonomously. In practice, of course, this also means that an IoT device needs access to the Internet at its location, which is not a problem in most cases. Most providers integrate their IoT devices into existing WiFi networks that have a connection to the Internet. The pro- vider of the respective software has to take care of the rest. Foundries.io is up to pace here, too. Every part of the operating system – from the bootloader to the kernel to userspace – can easily be updated remotely according to defined stan- dards. From a security point of view, this is a smart implementation. If you implement IoT devices with Foundries.io, you can, for example, 28 ADMIN 70 WWW.ADMIN-MAGAZINE.COM

Foundries.io TOOLS specify that updates always need a Alliance Lightweight Machine-to- other. Decoupling is also useful digital signature for the target de- Machine (OMA LwM2M) protocol, from a development point of view, vices to import them. In this way, a which specifies several standard com- as well as in everyday operation, provider can prevent attackers from munication methods between IoT because it enables far more granular hijacking its devices by loading a devices, such as message queuing te- work than monolithic firmware. hacked firmware update. For user- lemetry transport (MQTT) over HTTP. space updates, Foundries.io relies The following basically applies: Once Additional Functions on libostree (previously OSTree) to a protocol has established itself as a update individual parts right down standard in the IoT environment, the Up to this point, my main focus has to individual files, ensuring that up- chances are good that the Foundries been on the Foundries stack (i.e., the dates can be installed regularly and stack will support it. distribution for embedded systems incrementally instead of in the form that forms the core of the Foundries.io of major release cycles. Enter Docker portfolio). However, the provider’s ser- vice is not limited merely to offering a Beyond IoT The Foundries stack comes with ready-for-use image of this distribution. Docker as the runtime engine for Instead, Foundries Factory presents The examples used in this article containers, and third-party ven- itself as a full-blown development en- so far refer to devices from the IoT dors are advised to integrate their vironment that can be used to produce world, which are more likely to be specific customizations into the versions of the embedded distribution found in a domestic environment. Foundries framework in the form that are highly adapted to individual However, IoT now also plays a major of Docker containers for several devices. role in industry, and many func- reasons. Like the rest of the system, The first step for companies that want tions for this purpose can also be Docker containers in the Foundries to use Foundries Factory is therefore found in the Foundries stack. If you stack can be upgraded to new ver- likely to be creating an account on want to roll out IoT devices as edge sions with OTA updates. Therefore, the provider’s website. Doing so cre- applications in the enterprise, for vendors can effectively eliminate ates an entire development environ- example, you might need an option errors in IoT devices – even in the ment in which the Foundries stack is to execute commands directly from vendor-specific software compo- already present as a core component. a safe distance. The developers have nents. Moreover, companies benefit Of course, none of this just falls into implemented this scenario by includ- from clear demarcation between the place accidentally. The Foundries de- ing the WireGuard virtual private operating system on the one hand velopers primarily rely on Yocto [2]; network (VPN) client. If configured and their own applications on the in turn, Yocto dynamically generates appropriately, systems with the Foundries stack open a VPN connec- Figure 4: Foundries Factory offers comprehensive fleet management. Embedding a corre- tion to their home address and are sponding function in your firmware means you can ensure that remote devices automatically then open to receiving instructions. register with Factory as soon as they have an Internet connection. © Foundries.io The native cloud connectors for the major hyperscalers that come with the product are also aimed at business customers. For example, services and instances on AWS or Azure can be started or manipulated from the system without you having to implement the interface to these cloud environments yourself. Again, Foundries.io saves you considerable overhead by including a ready-made solution for a standard task. The same applies to support for the protocols typically found in the IoT environment, which is included by default. For example, if the device has a chip with Zigbee support (Zigbee being the protocol for controlling smart lighting), Foundries.io provides an interface for it in the system. The same applies to the Open Mobile WWW.ADMIN-MAGAZINE.COM ADMIN 70 29

TOOLS Foundries.io a development environment for em- the target systems download it gradu- In particular, the product is aimed bedded devices on the basis of the ally while the Internet connection is OpenEmbedded project. up and running. at companies that want to open up In this environment, companies then If you want to sell your customers re- have the ability to customize the ven- mote maintenance and management IoT options for their devices virtu- dor’s generic embedded distribution in addition to IoT devices, Foundries to suit the IoT device of their dreams. Factory again has all the necessary ally from scratch, without having to Foundries Factory delivers bootable components. Management capabili- images at the push of a button, and ties are an essential part of the offer- develop a basic system themselves. the images can be forwarded to a ing, precisely because they efficiently company’s hardware manufacturer, enable long-term management of Foundries.io provides a complete tool- who can then install the software di- delivered devices (Figure 5). rectly on the devices. The vendor sup- box from which suitable applications ports developers with, for example, Not Cheap, But Inexpensive native Git integration in Foundries can be easily assembled. In addition Factory or by providing a complete The Foundries.io pricing model dif- continuous integration/continuous fers from the usual approach to such to a full-blown development environ- deployment (CI/CD) build chain. solutions in one important respect: Once the devices reach the cus- Foundries.io charges flat rates and ment, it contains a ready-made mini- tomer’s data center or living room, does not seek to make money on Foundries Factory offers comprehen- every supported device. There is also distribution based on the Linux ker- sive fleet management (Figure 4), no such thing as a complete product which is more or less the server side structure with different editions. nel with a runtime environment for of the OTA update process: When the Instead, you have exactly one op- vendor of a device provides an up- tion – the Enterprise Factory package, Docker containers. These tools can dated image for a single component which costs $5,000 per month or of the system from Foundries Factory, $50,000 per year and includes all the significantly reduce a manufacturer’s described func- time to market for an IoT device. tions for an un- limited number Commercial use of Foundries Factory of devices and unlimited num- should be well worth their while in bers of builds in the Foundries most cases. Factory CI/CD environment. Of Technically, the solution is cutting edge. course, this also means that the Extensive support for a large number more devices a provider sells of security functions is just as useful with this soft- ware, the more as the cleverly thought-out update pro- sense it makes to use the solution. cess, which allows all the components Conclusions of an entire fleet of IoT devices to be replaced individually. Against this background, the manufacturer’s prices might not look cheap, but they are in- expensive in relation to performance, especially when you consider that Foundries.io, unlike other manufactur- ers, offers fixed prices and does not look to earn money on every unit of a device that is sold. Anyone who needs a powerful envi- ronment for developing and running IoT applications definitely needs to check out Foundries.io. Happily, it is quite simple to try Foundries out: The test account is available free online, and because Foundries Fac- tory offers support for a variety of SoC boards with an ARM or Intel CPU, a single Raspberry Pi is all you need to test the deployment on hardware. Q Figure 5: OTA updates are essential for IoT devices. The Foundries Foundries Fac- Info stack provides the ability to replace components with a newer tory proves that [1] Supported boards: version at any level of its architecture. © Foundries.io IoT devices do not have to be- [https://docs.foundries.io/latest/reference- come obsolete manual/linux/linux-supported.html] shortly after [2] Yocto: [https://www.yoctoproject.org] delivery and therefore do not The author have to pose a Freelance journalist Martin Gerhard Loschwitz security risk. focuses primarily on topics such as OpenStack, Kubernetes, and Chef. 30 ADMIN 70 WWW.ADMIN-MAGAZINE.COM



TOOLS PowerDNS and MariaDB PowerDNS Authoritative server high availability with MariaDB Galera Power Up Combining the PowerDNS Authoritative server daemon with MariaDB’s multiprimary Galera cluster allows a simple yet robust solution for your DNS needs. By Donnie Greer Recently, I found myself in the need fails, it is entirely possible that the both articles to get your Authoritative Lead Image © Sergey Nivens, 123RF.com for a trio of Authoritative nameserv- replica might not have recorded the server into a solid configuration. ers to disperse between my com- last few transactions. With a transac- Assuming PowerDNS is installed and pany’s data centers. Having used a tion-safe engine, such as InnoDB, a configured with the MySQL back end, PowerDNS Recursive server for years, transaction will either be completed you should crunch away on installing I was anxious to give their Authorita- on replica nodes or not at all. That MariaDB. The first step is to download tive version a heaping helping of DNS just won’t do. and run the MariaDB repo setup script: records. Enter MariaDB’s Galera Cluster. Unlike PowerDNS Recursive, the Galera is a virtually synchronous wget https://downloads.mariadb.com/U Authoritative server requires a back- multiprimary cluster for MariaDB MariaDB/mariadb_repo_setup end system to store records. The that is only available on Linux and list of supported back ends is rather only supports the InnoDB engine chmod +x mariadb_repo_setup lengthy, including but not limited to for storage (although MyISAM and sudo ./mariadb_repo_setup U MySQL, PostgreSQL, Berkeley Inter- Aria are in the works). With Galera, net Name Domain (BIND), and even you get virtually synchronous rep- --mariadb-server-version=\"mariadb-10.5\" Lightweight Directory Access Proto- lication, active-active multiprimary col (LDAP). I consider myself rather topology, read/write to any node, MariaDB documentation recommends skilled at MariaDB, and because automatic membership control, au- installing dependencies separately to the Authoritative server supports tomatic node joining, true parallel avoid conflicting packages from your MySQL, I knew that MariaDB would replication, and direct client con- OS vendor: be a non-issue. nections. Those features translate to I’ve set up dozens of MySQL rep- no replica lag, no lost transactions, sudo dnf install perl-DBI libaio U lication servers over the years, but read scalability, and smaller client libsepol lsof boost-program-options I wanted to investigate something latencies – perfect for keeping DNS different, something a bit better records happy and healthy across sudo dnf install U suited to this project’s specific needs. data centers. --repo=\"mariadb-main\" MariaDB-server MySQL Replication uses a primary server to update one or more replicas, Installation Once dependencies have been ad- and because the transactions are com- dressed, run the installation com- mitted sequentially, a slow transac- PowerDNS installation and setup has mand for MariaDB and start the Mari- tion can cause replicas to trail behind been covered at length in previous aDB server with systemctl: the primary server. If the primary articles [1] [2]. I highly recommend sudo mysql_install_db sudo systemctl start mariadb.service sudo mysql_secure_installation 32 ADMIN 70 WWW.ADMIN-MAGAZINE.COM

PowerDNS and MariaDB TOOLS The final line ensures the use of basic Listing 1 shows a quick user addition The basic configuration is complete, security best practices. to the database and the granting of but you have a bit more to do. MariaDB permissions. To exit from the Mari- needs to know that you intend to use Creating and Populating aDB shell, use the quit; command. Galera to cluster your PowerDNS da- For PowerDNS to work as intended, tabase. To do that, edit /etc/my.cnf.d/ A quick edit of your pristine Mari- you need to add the default schema server.cnf and add or modify the vari- aDB server’s configuration file /etc/ (Listing 2 for PowerDNS 4.3). Sche- ables shown in Listing 3. Pay special my.cnf.d/server.cnf (1) binds the mas for PowerDNS version 4.2 or 4.1 attention to the wsrep_cluster_address MariaDB service to the localhost can be found on the PowerDNS docu- variable because it is the list of IP ad- and (2) connects and (3) creates the mentation website [3] [4]. dresses of all nodes in the cluster. To database: Listing 1: User Creation Bind-address = 127.0.0.1 mysql -h localhost -u root -p GRANT ALL ON pdns.* TO 'pdnsadmin'@'localhost' IDENTIFIED BY 'CreateAnAwesomePassword'; create database pdns; GRANT ALL ON pdns.* TO 'pdnsadmin'@'localhost.localdomain' IDENTIFIED BY 'CreateAnAwesomePassword'; FLUSH PRIVILEGES; Listing 2: Populate a Database 01 use pdns; 43 name VARCHAR(255) NOT NULL, 02 44 type VARCHAR(10) NOT NULL, 03 CREATE TABLE domains ( 45 modified_at INT NOT NULL, 04 id INT AUTO_INCREMENT, 46 account VARCHAR(40) CHARACTER SET 'utf8' DEFAULT NULL, 05 name VARCHAR(255) NOT NULL, 47 comment TEXT CHARACTER SET 'utf8' NOT NULL, 06 master VARCHAR(128) DEFAULT NULL, 48 PRIMARY KEY (id) 07 last_check INT DEFAULT NULL, 49 ) Engine=InnoDB CHARACTER SET 'latin1'; 08 type VARCHAR(6) NOT NULL, 50 09 notified_serial INT UNSIGNED DEFAULT NULL, 51 CREATE INDEX comments_name_type_idx ON comments (name, type); 10 account VARCHAR(40) CHARACTER SET 'utf8' DEFAULT NULL, 52 CREATE INDEX comments_order_idx ON comments (domain_id, modified_at); 11 PRIMARY KEY (id) 53 12 ) Engine=InnoDB CHARACTER SET 'latin1'; 54 CREATE TABLE domainmetadata ( 13 55 id INT AUTO_INCREMENT, 14 CREATE UNIQUE INDEX name_index ON domains(name); 56 domain_id INT NOT NULL, 15 57 kind VARCHAR(32), 16 CREATE TABLE records ( 58 content TEXT, 17 id BIGINT AUTO_INCREMENT, 59 PRIMARY KEY (id) 18 domain_id INT DEFAULT NULL, 60 ) Engine=InnoDB CHARACTER SET 'latin1'; 19 name VARCHAR(255) DEFAULT NULL, 61 20 type VARCHAR(10) DEFAULT NULL, 62 CREATE INDEX domainmetadata_idx ON domainmetadata (domain_id, kind); 21 content VARCHAR(64000) DEFAULT NULL, 63 22 ttl INT DEFAULT NULL, 64 CREATE TABLE cryptokeys ( 23 prio INT DEFAULT NULL, 65 id INT AUTO_INCREMENT, 24 disabled TINYINT(1) DEFAULT 0, 66 domain_id INT NOT NULL, 25 ordername VARCHAR(255) BINARY DEFAULT NULL, 67 flags INT NOT NULL, 26 auth TINYINT(1) DEFAULT 1, 68 active BOOL, 27 PRIMARY KEY (id) 69 published BOOL DEFAULT 1, 28 ) Engine=InnoDB CHARACTER SET 'latin1'; 70 content TEXT, 29 71 PRIMARY KEY(id) 30 CREATE INDEX nametype_index ON records(name,type); 72 ) Engine=InnoDB CHARACTER SET 'latin1'; 31 CREATE INDEX domain_id ON records(domain_id); 73 32 CREATE INDEX ordername ON records (ordername); 74 CREATE INDEX domainidindex ON cryptokeys(domain_id); 33 75 34 CREATE TABLE supermasters ( 76 CREATE TABLE tsigkeys ( 35 ip VARCHAR(64) NOT NULL, 77 id INT AUTO_INCREMENT, 36 nameserver VARCHAR(255) NOT NULL, 78 name VARCHAR(255), 37 account VARCHAR(40) CHARACTER SET 'utf8' NOT NULL, 79 algorithm VARCHAR(50), 38 PRIMARY KEY (ip, nameserver) 80 secret VARCHAR(255), 39 ) Engine=InnoDB CHARACTER SET 'latin1'; 81 PRIMARY KEY (id) 40 CREATE TABLE comments ( 82 ) Engine=InnoDB CHARACTER SET 'latin1'; 41 id INT AUTO_INCREMENT, 83 42 domain_id INT NOT NULL, 84 CREATE UNIQUE INDEX namealgoindex ON tsigkeys(name, algorithm); WWW.ADMIN-MAGAZINE.COM ADMIN 70 33

TOOLS PowerDNS and MariaDB Listing 3: Galera Config More Power with HAProxy [6] and Keepalived [7]. [galera] Once the initial node is set up, sim- Imagine having a single web interface wsrep_on=ON ply follow the steps above two more wsrep_cluster_name=MyHappyLittleCluster times to create secondary and ter- that could update all PowerDNS Au- wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so tiary nodes. Remember to bootstrap wsrep_cluster_address=gc only the first node in your cluster thoritative servers at once, thanks to omm://192.168.0.10,192.168.0.20,192.168.0.30 and verify that /etc/my.cnf.d/ binlog_format=row server.cnf is identical on all three. Galera [8]. Q innodb_autoinc_lock_mode=2 The systemctl command starts the other nodes after the Primary Com- Info add more nodes, simply add their IP ponent node. [1] “Speed up Your Name Server with a MySQL address separated by a comma. Take note that the variables shown Testing Back End” by Joseph Guarino: in Listing 3 are the minimal, manda- [https://www.admin-magazine.com/ tory variables to make Galera a happy After adding more nodes to the clus- Articles/Speed-up-Your-Name-Server- camper, but many more tunable good- ter, a quick and easy test is to connect with-a-MySQL-Back-End/] ies can be found in Galera Variables to any node in the cluster with the [2] “Exploring PowerDNS” by Joseph Gua- documentation [5]. MariaDB client and run a quick SQL rino: [https://www.admin-magazine. Additionally, the new cluster nodes statement: com/Articles/PowerDNS-The-Other- will attempt to connect to other Open-Source-Name-Server/] nodes listed in wsrep_cluster_address sudo mariadb [3] PowerDNS 4.2 schema: in search of a Primary Component, SHOW GLOBAL STATUS U [https://github.com/PowerDNS/pdns/blob/ which will be the first node you boot- rel/auth-4.2.x/modules/gmysqlbackend/ strap with the galera_new_cluster LIKE 'wsrep_cluster_size'; schema.mysql.sql] script. To bootstrap a new cluster and [4] PowerDNS 4.1 schema: create a new Primary Component, If the size of the cluster is equal to [https://github.com/PowerDNS/pdns/blob/ run the command the number of nodes in the cluster, rel/auth-4.1.x/modules/gmysqlbackend/ it’s time to celebrate! If the size is schema.mysql.sql] sudo galera_new_cluster smaller than expected, either a node [5] MariaDB system variables and options: did not start correctly, or it cannot [https://mariadb.com/docs/deploy/ only on the first node. This com- connect to the Primary Component. community-cluster-cs10-5-rhel8/] mand identifies the first node as a [6] HAProxy: [http://www.haproxy.org/] “seed” to populate the databases of Conclusion [7] Keepalived: newly added nodes. Therefore, all [https://github.com/acassen/keepalived] nodes added to the cluster will auto- If you need basic and easy data repli- [8] PowerDNS-Admin: [https://github.com/ matically copy the complete schema cation with minimal setup time, Mari- PowerDNS-Admin/PowerDNS-Admin] and data without user intervention. aDB’s Galera Cluster has you covered. How cool is that?! Just think what you could do with a The Author slightly more complex configuration Donnie Greer is a 26-year IT veteran specializing in Linux, IaaS, and as little work as possible. He can often be heard yelling at his Synology NAS: “Work faster! I believe in you!” Q 34 ADMIN 70 WWW.ADMIN-MAGAZINE.COM



TOOLS Dogtag Certificate management with FreeIPA and Dogtag Show Your ID The Dogtag certificate manager integrated into the FreeIPA open source toolset generates SSL/TLS certificates for intranet services and publishes them on the network. By Andreas Stolzenberger Both internal and external services All you need is your own certificate management anyway, just as admins Photo by Oscar Sutton on Unsplash rely on encrypted communication authority (CA) on your intranet to run AD on Windows networks. with SSL and TLS. For external ser- manage and sign certificates for the vices, administrators use officially connected services. Internal comput- Installing the FreeIPA signed certificates, although Let’s En- ers then only need to trust that this Server crypt is absolutely fine in many sce- internal root CA for all keys signed narios. In contrast, internal services by it are identified as valid. To set up the IPA server, you will predominantly rely on self-signed Dogtag [1], the open source certificate want to use a Linux distribution certificates, which always cause a stir system, offers a simple approach to such as Enterprise Linux distribu- with web browsers on the local area managing an internal CA, and it inte- tion version 8 (EL8), Rocky, or Al- network (LAN) by generating mes- grates seamlessly with the FreeIPA [2] maLinux. The Minimal installation sages such as The server’s certificate user directory. FreeIPA is to Linux type is quite sufficient. The setup is unknown. what Active Directory (AD) is to the does not need special repositories Administrators would prefer to see Windows world. It uses the same such as Extra Packages for Enter- a nice lock icon displayed in the technology with a Lightweight Direc- prise Linux (EPEL) because all of browser for a trusted TLS connec- tory Access Protocol (LDAP) back end the required packages can be found tion – for their intranet applications, and Kerberos authentication. AD and in the AppStream repository of ev- too – instead of requiring users to an Identity, Policy, and Audit (IPA) ery EL8 distribution. The IPA server create an individual exception in the system can trust each other with cross- must have a static IP address. Its browser for every internal applica- domain trusts, allowing administrators Domain Name System (DNS) name tion. This also means that stricter of heterogeneous networks to run a must be configured correctly, and security policies can be applied for connected directory for Windows and DNS resolution on the LAN must browsers on the corporate network, Linux machines. be correct (forward and reverse preventing users from opening un- In this article, I review the basic fea- lookup). The time zone configura- trusted connections at all or from tures of FreeIPA and focus on Dog- tion must be appropriate, and the creating exceptions. Admins also tag. Every administrator of an envi- server must use Chrony to sync the want other internal services to use ronment with multiple Linux servers time with timeservers on the web. trusted certificates and SSL for com- will probably run an IPA directory The FreeIPA server is registered munication. for central user, host, and service as a module named idm (identity 36 ADMIN 70 WWW.ADMIN-MAGAZINE.COM

Dogtag TOOLS management) in the AppStream re- A Berkeley Internet Name Listing 1: dnsmasq Output pository. The versions you will find Domain (BIND) 9 DNS server are simply the server and the client; needs the same entries, but in srv-host =_kerberos._udp.mykier.ip,ipa.mykier.ip,88 the dandified Yum (dnf) command a different format: srv-host =_kerberos._tcp.mykier.ip,ipa.mykier.ip,88 lets you enable the server module and srv-host =_kerberos-master._tcp.mykier.ip,ipa.mykier.ip,88 set up the required packages: _kerberos._udp.mykier.ip. 86400 U srv-host =_kerberos-master._udp.mykier.ip,ipa.mykier.ip,88 dnf module enable idm:DL1 IN SRV 0 100 88 ipa.mykier.ip. srv-host =_kpasswd._tcp.mykier.ip,ipa.mykier.ip,88 dnf distro-sync srv-host =_kpasswd._udp.mykier.ip,ipa.mykier.ip,88 dnf module install idm:DL1/server [...] The installation is an interactive pro- The important thing here is srv-host =_ldap._tcp.mykier.ip,ipa.mykier.ip,389 cess with the that both AD and IPA require ipa-server-install the same service (SRV) records for certificates issued (Figure 1). For fur- command. Alternatively, you can pass all the required parameters into Kerberos in DNS. If you are already ther tests with the directory, you can the command at the command line. At this point, you need to decide using AD on the LAN, do not reas- now create users and groups. With whether the IPA server itself will as- sume the role of DNS server, which sign the entries; otherwise, AD will the use of suitable rule sets (e.g., is common practice with AD. In this example, however, I assume that a no longer work. FreeIPA can manage Sudo rules), you can allow individual DNS installation is already in place on your network. without a DNS configuration. How- users or groups to work as root users To make it easier for IPA clients and servers to communicate, you need to ever, you then always need to include on selected hosts. add a few entries to the existing DNS server that point to the IPA installa- the reference to the realm and the IPA To add an existing Linux server to the tion. In this example, the server is named ipa.mykier.ip and it manages server in all operations on the clients. directory, you first need to install the the mykier.ip realm. Running dnsmasq reveals the matching entries in the To enable the clients to communicate IPA client on the server. As for the IPA DNS server (Listing 1). with the IPA server, you also need to server, you will find the packages you open the required firewall ports on need in the AppStream repository: the IPA machine: dnf install @idm:client firewall-cmd --add-service={U http,https,dns,ntp,freeipa-ldap,U Once you have configured the IPA freeipa-ldaps} --permanent entries on your DNS server, a simple firewall-cmd --reload command is all it takes to register the server in the domain: You can now access the web user interface (UI) on the FreeIPA server ipa-client-install --mkhomedir at https://ipa.mykier.ip. When you get there, you will see the users and The installer relies on DNS to dis- hosts in your domain along with any cover all the information it needs Figure 1: FreeIPA listing all the issued certificates. ADMIN 70 37 WWW.ADMIN-MAGAZINE.COM

TOOLS Dogtag about the domain. The --mkhomedir If you use Kickstart to install EL8 authorized with admin rights to the option tells the client to create the servers or clients, this step can be au- directory: required home directories dynami- tomated. If you are setting up Fedora cally for the domain users on the Linux or EL8 Linux with a graphical ipa host-add hostname.fqdn U local system. In practice, however, UI (GUI) from a Live operating system --password=<one-time-password> administrators are more likely to cre- disc, you can handle the domain join- ate an automount rule on the FreeIPA ing step directly in the Gnome con- The assigned password expires after server that mounts the home directo- figuration after completing the basic the computer’s first attempt to join ries from a central NFS server at user installation. When Gnome prompts the domain. login time. In this way, domain users the user to create a new user, the En- On existing Linux hosts, install the will always have access to their files, terprise Login button pops up, which IPA client Ansible or some other au- regardless from which PC within the then guides you interactively through tomation tool. Then, on the respec- domain they log in. the IPA client setup. tive client, by remote execution (or Without DNS records, users need to an Ansible role), register with the pass in the IPA server and domain in- Automating the IPA Client domain: formation to the command: Rollout ipa-client-install U ipa-client-install U When you set up a FreeIPA directory --hostname=client.mykier.ip U --mkhomedir U on an existing LAN, you obviously --domain=mykier.ip U --server=ipa.mykier.ip U don’t want to have to register manu- --mkhomedir -w <one-time-password> U --domain=mykier.ip U ally all existing or new machines --realm=MYKIER.IP U --realm=MYKIER.IP with the directory. The process can --server=ipa.mykier.ip be automated in a relatively simple In both cases, the client setup then way. First, create the appropriate You only need the details (e.g., do- asks for a domain admin user account host entries in the directory ser- main, server, and realm) if you do not name and password to log the current vice, which you can do on a com- have appropriate DNS records for the machine into the directory service. puter that has the IPA client and is IPA server. By the same principle, you can automatically add systems installed by Kickstart to the directory. First define the computer, including the one-time password in the direc- tory. In the Kickstart file, add the fol- lowing options: %packages [...] ipa-client and toward the end: %post [...] /usr/sbin/ipa-client-install U <parameters as above> Figure 2: Sudo rules specify with which directory users can assume the root role on the The system then joins the domain at various target systems. The rules can also be limited to selected commands. the end of the automated installation process. In the web UI of your IPA server, the newly registered server is now displayed in Identity | Hosts. With Policy | Sudo | Sudo rules you could now, for example, allow one of your directory users to run the sudo com- mand on the new server and perform actions as root. The ruleset also sup- ports granular breakdown if you only 38 ADMIN 70 WWW.ADMIN-MAGAZINE.COM

Dogtag TOOLS want to let users run specific com- EL8 systems run SELinux by default. qualified domain name (FQDN) of the mands or command groups as Sudo The directory needs the correct context server multiple times, you can define it without gaining full root access to the for the certificates; otherwise, the IPA as a shell variable named $fqdn. The system (Figure 2). client cannot store certificates there: ipa service-add Generating Certificates for semanage fcontext -a U Services -t cert_t \"/etc/nginx/cert(/.*)?\" command registers the desired service for the server against the directory. Once the new server has joined the restorecon -v /etc/nginx/cert A number of services are predefined domain, you can create certificates (e.g., HTTP), but you can create your for services on this machine and have In the next step, log in to the direc- own services at any time (e.g., REDIS/ them signed by the root CA on the Free tory with an admin account (kinit redis.mykier.ip). IPA server (Figure 3). In this example, admin). Now go to the certificate The command I run a web server with Nginx on a directory, register the service, and freshly set up AlmaLinux 8 server that request the appropriate certificate. ipa-getcert request I registered in the domain as dhcp224. I For this case study, assume that the want to give this server an official cer- realm for your domain is MYKIER.IP: creates the request and also im- tificate for the HTTPS protocol: mediately intercepts the response cd /etc/nginx/cert from the Dogtag service on the IPA dnf install nginx fqdn=$(hostname -f) server. Like many other services, ipa service-add HTTP/$fqdn Nginx wants the certificate in pri- Now create a suitable directory for ipa-getcert request -f $fqdn.crt U vacy-enhanced mail (PEM) format the certificate in two files for the certificate and -k $fqdn.key -r U the key. If the server is named www. mkdir -p /etc/nginx/cert -K HTTP/[email protected] -N $fqdn mykier.ip, you will receive a www. To avoid typing the complete fully

TOOLS Dogtag mykier.ip.crt file and a www.mykier. Building Trust Relations On an EL8 or Fedora Linux system, ip.key file in the response. The long copy the ca.crt from your IPA server to detour by a local Network Security For your connected clients to accept the /etc/pki/ca-trust/source/anchors Services (NSS) database is no longer the certificates created in this way, directory (don’t forget the SELinux needed thanks to Certmonger. When they need to trust the CA on your IPA context) and run the update-ca-trust a certificate is approaching its expi- server, where you will find the root command as root or with sudo. Applica- ration date, you can renew it with CA in /etc/ipa/ca.crt. The method tions such as the Chrome or Chromium the -r (renew) option. you need to import a custom CA de- browser use the system’s trust chain For the Nginx server to access the pends entirely on the application. For and do not need an individual configu- files, it of course needs the appropri- example, the Firefox browser lets you ration. On Windows, you can import ate permissions: store ca.crt in a specific directory. On your own certificates with a Group Windows, this would be Policy or a PowerShell script. chown -R nginx. /etc/nginx/cert/* - \"%USERPROFILE%\\AppData\\Local\\U Conclusions In the default /etc/nginx/nginx.conf Mozilla\\Certificates\" configuration file, simply enable FreeIPA with the integrated Dogtag the commented out # Settings - \"%USERPROFILE%\\AppData\\Roaming\\U services and Certmonger client greatly for a TLS enabled server section Mozilla\\Certificates\" simplifies certificate management on and add the reference to your the intranet. Gone are the days of local certificate: and on Linux: NSS databases and tedious workflows to copy requests and their responses ssl_certificate U - \"/usr/lib/mozilla/certificates\" back and forth – along with the need to \"/etc/nginx/cert/www.mykier.ip.crt\"; - \"/usr/lib64/mozilla/certificates\" convert the resulting PKS12, PEM, KEY, or CRT files somehow to the right for- ssl_certificate_key U On Linux systems with SELinux mats. The open source toolset provides \"/etc/nginx/cert/www.mykier.ip.key\"; enabled, it is again important for you a quick option for creating and de- the certificates directory and the ploying certificates for all TLS-enabled After restarting the service, your contained files to have the correct services in just a few simple steps. Q web server now also responds cert_t context; otherwise, Firefox to HTTPS requests with the IPA- cannot read them. Applications such Info signed certificate. You can then use as Redis either pass the path in to [1] Dogtag: [https://www.dogtagpki.org/wiki/ the same principle to create other ca.crt at the command line or save services (e.g., IMAP, SMTP, or it in the configuration file. Today, PKI_Main_Page/] even MQTT) and generate suitable many applications use the operating [2] FreeIPA: [https://www.freeipa.org/page/ certificates. system’s CA trust. Main_Page] Figure 3: Issuing certificates for individual services and machines. WWW.ADMIN-MAGAZINE.COM 40 ADMIN 70



TOOLS Prometheus Anomaly Detector Detect anomalies in metrics data Jerk Detector Anomalies in an environment’s metrics data are an important indicator of an attack. The Prometheus time series database automatically detects, alerts, and forecasts anomalous behavior with the Fourier and Prophet models of the Prometheus Anomaly Detector. By Martin Loschwitz Attacks on environments are just enabling you to act as quickly as pos- what an anomaly is in the context of Lead Image © rudall30, 123RF.com as much a part of the daily grind in sible in a specific case (i.e., conveying a particular environment. Large in- IT as operating the IT infrastructure the current situation). frastructures, for example, will apply itself. The range of attacks is wide This approach is not particularly up far higher thresholds for DDoS than and depends on the attacker’s goals. to date or smart. Modern monitoring websites with only a few visits per Classic denial-of-service attacks are systems like Prometheus collect such day. From your point of view, anomaly not complex and quite easy to detect. large volumes of metrics data that it detection now means finding a reliable However, when the focus shifts to can be used to identify trends and mean value for individual datapoints sniffing data, the methods are far anomalies, potentially indicating that and then defining limits within which more subtle, and highly complex IT attacks are in progress. Even distrib- the current measured values are al- attacks on different levels are no lon- uted denial-of-service (DDoS) attacks lowed to deviate from the norm. The ger challenging. have ceased to follow the principle of “cry wolf” effect of permanent false As complex as the attack scenarios taking a server offline with as much positives should not be underesti- are, one factor remains the same: Ad- traffic as possible in as short a time mated. Sooner or later, no one will ministrators want to notice as early as as possible. Instead, postmortem take a monitoring system seriously if possible that bad things are going on in analyses of attacks regularly reveal it constantly sounds the alarm without their setups so they can react promptly. that attackers successively increased reason. Instead of a blunt weapon, The sooner an attack is detected, the the traffic in the weeks leading up to a fine scalpel comes into play when sooner it can be counteracted and the an attack and did so in such a way detecting anomalies in metrics data, less damage it can cause. that they always flew under the ra- and the Z-score is a prime example of dar of the thresholds in monitoring. a particularly good scalpel. Rigid Limits of Limited Use At the decisive moment, a relatively A little excursion into the world of small peak in the attack volume was Carl Friedrich von Gauss’s mathemat- The ability to detect an attack early the final straw that broke the serv- ics is unavoidable. Most people have depends on the tools available and ers’ backs. With better trend analysis probably heard of Gaussian normal how you use them. In the past, most (e.g., with the help of Prometheus), distribution. Simply put, Gaussian admins relied on run-of-the-mill event such attacks become quite predict- theory states that, for any number of monitoring with thresholds: If the able. measured values, the extremes occur incoming data volume exceeded a rarely and the median (i.e., the 50th certain limit, the monitoring system Gaussian Z-Scores percentile) occurs particularly fre- sounded an alarm. If too many in- quently. On both sides of the x-axis, valid login attempts appeared in the The statistical Z-score plays an impor- the number of matches per value in- servers’ authentication logfiles, you tant role when it comes to detecting creases as the median is approached. were notified. The focus here is on anomalies, allowing you to define Given 100 servers, the power 42 ADMIN 70 WWW.ADMIN-MAGAZINE.COM

Prometheus Anomaly Detector TOOLS consumption of most devices is likely generally assumes that you will roll now considered a standard algorithm to fall around the median, with a few out PAD as a component in Open- of modern IT. The Fourier module individual machines requiring par- Shift, although this scenario is not in Python has an AI-trainable model ticularly greater or very little power. enforced. However, if you want to use to predict the further evolution of an These values form the outer extremes PAD, you will need an environment existing graph on the basis of histori- of an imaginary chart with all mea- comprising Prometheus, Alertman- cal values. sured datapoints. ager, and Thanos – more about this The other algorithm that PAD uses Percentiles generally play an impor- in a moment – because PAD does not to detect anomalies, Prophet, comes tant role in calculating the Z-score. seek to be a monitoring tool itself, but directly from the Facebook social net- The first step is to calculate the to dock onto existing setups. At its work (Figure 1) and is significantly median, which is the 50th percen- core, PAD is a Python application that more complex in direct comparison tile (i.e., 50 percent of all measured applies two Python libraries for arti- with Fourier. In return, it allows fac- values correspond to this value). The ficial intelligence (AI) and machine toring into its predictions such things Z-score is used to find out how far learning (ML): Fourier and Prophet. as the potential seasonality of data, a single datapoint deviates from this including the factors year, week, and median and is calculated by Fourier and Prophet day. All told, the dataset that Prophet uses to analyze ongoing data streams, which can be either defined with Metrics are available as numbers in predict their continuation, and raise generic values or determined indi- Prometheus; however, sinusoidal the alarm if necessary is far larger. vidually. Common values are the 68th curves are far better suited for ac- PAD, with its two implementations percentile (i.e., in a dataset of 100 curate analysis of the current data, of anomaly detection, goes signifi- values, 68 of those fall within +/-1 as well as the future development cantly further than the analysis of the standard deviation [SD] of the mean), of these values in the context of acquired metrics described at the be- the 95th percentile (+/-2SD), and the machine learning. Each value is rep- ginning. The aim is not just to notice 99.7th percentile (+/-3SD). resented in the form of a frequency. that something is wrong at an early The Fourier algorithm is responsible stage, but to notice it even earlier: Red Hat with Groundwork for the transformation between the Both Fourier and Prophet are trainable worlds by using sine and cosine to AI models in PAD that use existing Now the question arises as to what convert the numbers into frequency metrics data to predict the evolution you need to do to generate appropri- signals, after which useful basic data of the respective metrics, allowing ate alerts from your metrics data, is available for the prediction. This for a response at the slightest sign of which is a practical possibility only if process is known as “fast Fourier an anomaly. Your task is to compare you use a time series database (e.g., transformation,” which goes back the developments of a metric value Prometheus). Prometheus collects to the mathematicians James Cooley calculated by Prophet or Fourier with the metric values from “exporters” on and John Tukey, who popularized the actual state. Ideally, thanks to the the target systems and stores them the idea for converting data into trained models in PAD, you will notice centrally. This data can be evaluated sinusoidal curves in a paper that ap- very quickly that something suspicious by a custom query language, and peared in 1965 [1]. The method is is going on (Figure 2). Grafana can display the Prometheus data graphically. Prometheus gener- Figure 1: The Prophet PAD component is trimmed to respond correctly with predictive ates alerts with its alert manager methods to the smallest deviations in metric data, represented here by the black dots. component if individual metrics as- sume certain values or are outside of defined limits. The question of how use an existing Prometheus installation for effective anomaly detection is provided by Red Hat. The Prometheus Anomaly Detec- tor (PAD) comprises several compo- nents designed to detect anomalies from historic data on the one hand and machine learning and projection on the other. A look under the hood shows the combination of components. Red Hat WWW.ADMIN-MAGAZINE.COM ADMIN 70 43

TOOLS Prometheus Anomaly Detector A Failure to Scale metrics for a particular host. What’s Second, a Thanos component is re- more: Grafana also needs to be con- sponsible for storing historical data From the very beginning, the Pro- figured in the same way and different in a meaningful way. To this end, metheus developers designed their queries need to access the different Thanos provides an interface, known work to ignore high availability in the data sources in Grafana. as StoreAPI, to its own storage imple- classical sense. Instead, you have to What’s almost worse, though: The mentation, as well as interfaces to run multiple instances of Prometheus more data Prometheus stores, the other databases. Thanos provides Pro- at the same time. Because retrieving slower and more unresponsive it metheus with the Sidecar component, metrics data from the target systems becomes. However, you do have a le- which dynamically handles writes ties up virtually no resources, it gitimate interest in retaining historical not to local Prometheus memory, doesn’t generate more load on the data because it allows for better scal- but distributed across the network. monitored systems. If one of the run- ability planning and makes it easier to The long-term storage of the his- ning instances fails, the idea is that identify trends. If you keep too much torical data is so much better than if you have enough other instances to legacy information in Prometheus, Prometheus itself were used for this query. In this way, comprehensive calling individual Grafana dashboards purpose. deduplication is implemented at the will soon take several seconds. In- In brief, Thanos is a practical exten- alert manager level: If 20 instances telligent downsampling of data is sion to the plain vanilla Prometheus feed an alert into the alert manager missing, as is high-performance stor- that has found its way into many in- because of the same metric value, age for archived data outside of Pro- stallations around the world. The fact the alert manager still sends only one metheus itself. that Prometheus anomaly detection message to the alerting targets. takes advantage of Thanos is hardly However, this use case turns ugly Thanos and Historical Data surprising: In particular, storing his- when it comes to horizontal scaling. torical data is extremely helpful when Massively scalable environments, in Thanos is now a separate project, it comes to detecting anomalies. particular those with hundreds or independent of Prometheus, that thousands of hosts, mean that indi- wraps around several self-sufficient PAD in Practical Use vidual Prometheus instances become Prometheus instances. First, Thanos a bottleneck over time. If you go for provides a unified query view: No As mentioned earlier, PAD is designed manual sharding, however, you lose matter which of the Prometheus in- to run in OpenShift, but the container the central advantage of the single stances connected to Thanos has the runs quite well without Red Hat’s or- point of administration because in- metrics data for a particular host, the chestrator. The following example as- dividual Prometheus instances then admin always talks to Thanos, which sumes that you use Podman to man- query a local list of targets, and it’s gathers the data appropriately in the age your containers. To begin, source your task to connect to the appropri- background. The same thing also ap- the complete PAD container; the code ate Prometheus instance to view the plies to Grafana instances. is also available on GitHub [2]: podman pull quay.io/aicoe/U prometheus-anomaly-detector:latest PAD gets the metrics for which it is supposed to detect anomalies per Fourier and Prophet via environment variables. Next, you need to build the command line to start the container. PAD gets the metrics for which it is supposed to detect anomalies with Fourier and Prophet from environment variables. These, in turn, can be easily passed in to the container. The command Figure 2: The blue line shows the predicted evolution of the data for the respective metric; docker run U yellow and green indicate the range of variation. The anomaly is clearly visible. --name pad -p 8080:8080 --network host U --env FLT_PROM_URL=U http://pad.local.lan:9090 U --env FLT_RETRAINING_INTERVAL_MINUTES=15 U --env FLT_METRICS_LIST='up' U 44 ADMIN 70 WWW.ADMIN-MAGAZINE.COM

Prometheus Anomaly Detector TOOLS --env APP_FILE=app.py U exporter. For this step, the develop- to allow for start-up time with false --env FLT_DATA_START_TIME=3d U ers rely on Flask. --env FLT_ROLLING_TRAINING_'WINDOW_SIZE=U The rest is plain sailing. Once the positives at the beginning. However, metrics data and the predictions are 15d quay.io/aicoe/U available in Prometheus, dashboards the extra work will pay off when you prometheus-anomaly-detector:latest can be created in Grafana in the usual way. What’s more, the predictions from track down an attacker because you starts the container and sets up as a Fourier and Prophet can be integrated metric in FLT_METRICS_LIST (i.e., you into the same dashboards and super- noticed even earlier than usual that want to know whether or not the sys- imposed – together with the measured tems are running). Instead of up, you values, if required (Figure 3). If you something was wrong. Q need to add the names of the metrics want to set up alarms from the predic- for which you want to detect anoma- tions, you can do so in the alert man- Info lies. If you enter the Prometheus ager. A Red Hat talk from 2019 [3] pro- [1] Cooley, J. W. and J. W. Tukey. An Algorithm Node Exporter’s node_filesys- vides some details of the configuration. tem_avail_bytes metric here, for ex- for the Machine Calculation of Complex ample, you are telling PAD to monitor Conclusions Fourier Series. Mathematics of Compu- changes in the allocation of storage tation, 1965;19:297-301, [https://www. drives of individual devices, because Administrators often turn up their ams.org/journals/mcom/1965-19-090/ suddenly increasing space occupation noses when vendors present their S0025-5718-1965-0178586-1/S0025-5718- (i.e., reductions in free space) can be AI solutions for attack detection; in 1965-0178586-1.pdf] an indicator of some undesirable pro- fact, it’s not uncommon to hear them [2] Prometheus Anomaly Detector: [https:// cesses that justify a closer look. referred to as hocus pocus. However, github.com/AICoE/prometheus-anomaly- PAD alone usually does not help Red Hat has come up with a very detector] you, because Prometheus does concrete and immediately usable ap- [3] AIOps: Anomaly detection with Pro- not visualize the data graphically. proach to generating added value in metheus, by Marcel Hild, Linux Founda- Grafana is the tool you want to use, everyday life with AI in the form of tion, [https://events19.linuxfoundation. and the PAD developers make it PAD. The more metric data Fourier org/wp-content/uploads/2017/12/ easy to do just that, because PAD and Prophet have available, the bet- AIOps-Anomaly-Detection-with- exports the calculated metrics data ter they can train their models and Prometheus-Marcel-Hild-Red-Hat.pdf] in Prometheus format. An existing the more reliable the predictions Prometheus instance can read the become. Therefore, you do not need The Author data from PAD just as from any other Freelance journalist Martin Gerhard Loschwitz focuses primarily on topics such as OpenStack, Kubernetes, and Ceph. Figure 3: Once the data from PAD has found its way into Prometheus, it can be visualized in Grafana, which is where you see a comparison between the data and the Prophet predictions. WWW.ADMIN-MAGAZINE.COM ADMIN 70 45

TOOLS Orchestration with Puppet Bolt Puppet Bolt orchestration tool Lightning Strike Puppet Bolt free software automates administrative tasks to speed up the admin’s daily work. By Holger Reibold Because Bolt is a member of the scripts can also be used to handle orchestration script wrapper, called a Photo by lee junda on Unsplash Puppet product family, the question complex tasks. “plan.” Above all, if statements are naturally arises as to how Bolt [1] dif- used in the scripts for concrete er- fers from Puppet. Puppet is used for Overview ror handling. Administrators who are continuous resource management. In already familiar with YAML files can particular, it provides monitoring func- Ad hoc commands and scripts are generate their tasks in this particular tionality and checks at short intervals run on the infrastructure with the format and then use Bolt’s built-in tool whether the services in question are Puppet Enterprise (PE) orchestrator to convert YAML files into Bolt plans. still available or whether infrastructure or with Puppet’s standalone task You will find various special features elements have gone missing. runner, Bolt. Bolt lets you patch beneficial: Bolt has pre-built scripts Bolt basically zooms in on point- and update systems and services, that you only need to adapt to spe- in-time changes. Instead of using troubleshoot servers, roll out appli- cific tasks; it also lets you use existing declarative statements that define cations, and start and stop services. automation scripts and offers support an infrastructure, Bolt is more about It runs on a standard workstation for Python, Ruby, and PowerShell. Al- when commands are executed and (Linux, Windows, macOS), and though many orchestration tools rely which ones. In particular, the tool secure shell (SSH), secure copy on agents, Bolt also supports agentless simplifies the execution or orches- (SCP), Windows Remote Manage- deployment or a combination of the tration of tasks. ment (WinRM), and other popular two strategies. Bolt also offers Bash Admins benefit from the ability authentication methods (password, support and workflow orchestration. to run a script over any number public key) connect to the remote of network nodes. Bolt uses plans node. According to the developers, Installation and Setup that bundle the execution details. the solution scales to more than The focus is particularly on error 1,000 simultaneous connections. Bolt runs on all popular operating handling, but comparatively simple Bolt uses YAML files or its own systems. In addition to a Linux-based 46 ADMIN 70 WWW.ADMIN-MAGAZINE.COM

Orchestration with Puppet Bolt TOOLS machine, you can use the orchestra- and run a Bolt cmdlet as a test. Ide- Bolt requires a specific directory tion tool on a macOS or Windows ally, you will not see any error mes- structure for the projects – plans and workstation. Installing Bolt on sages. If you do, you might need to tasks will not work without this struc- Debian is a matter of a few simple add more Bolt modules to PowerShell ture. In this context, the directory commands: or edit the execution authorizations. structure of a Bolt project is closely linked to Puppet modules. For exam- wget https://apt.puppet.com/U Task-Specific Configuration ple, if you plan to install Apache on a puppet-tools-release-bullseye.deb remote system, you need to create an Bolt offers a wide range of customiza- Apache module directory. sudo dpkg U tion options for global and project- To stay with this example, you need -i puppet-tools-release-bullseye.deb specific configuration. Four categories to generate a module directory along can be distinguished: with an Apache subdirectory by sudo apt-get update Q Customizing Bolt’s general behav- changing to the project directory and sudo apt-get install puppet-bolt creating two directories, one for the ior, such as choosing the format project files and one for the plans: To run Bolt on a macOS system, you for displaying the output and de- first need to install Homebrew [2], an fining the number of threads for cd first_bolt_project open source package manager for the connecting to targets mkdir -p module/apache/plans operating system. To install Home- Q Defining project-specific settings mkdir -p module/apache/files brew, run the following command in by specifying how to deal with the macOS terminal: concrete orchestration tasks, in- The directory structure for the first_ cluding configuring the path to bolt_project folder looks like this: /bin/bash -c \"$(U an inventory file or to a Hiera curl -fsSL U configuration file. (Hiera is a key/ |-- bolt-project.yaml https://raw.githubusercontent.com/U value database for the configura- |___module Homebrew/install/HEAD/install.sh)\" tion data.) Q Deciding which transport protocols |__apache Use the tap brew command to instruct to use, such as adjusting the path |- files Homebrew to use additional reposi- to your private SSH key or the port |_plans tories. By default, the tap command for the WinRM connection assumes you are accessing sources Q Grouping inventory data by targets Defining Targets from GitHub repositories, so you need and assigning them their own con- to prepare Homebrew for using the figurations The next step is to define the targets. Puppet sources by typing: Bolt options and functions are con- In this section, I assume that you figured at the project, user, or system want to run an Apache web server brew tap puppetlabs/puppet level. At the project level, you specify on a Docker installation and perform the Bolt configuration in the bolt-proj- typical tasks there. The connection To install Bolt, run the command: ect.yaml and inventory.yaml files. between the Bolt installation and the Customizations at the user and remote systems is set up with either brew install --cask puppet-bolt system level are defined in bolt-de- SSH or WinRM. As a rule, SSH will be faults.yaml. If the specific use case the better choice. To talk to a Docker Alternatively, you can use the macOS does not require user-specific or installation, generate a file named installer and use the DMG file from global configurations, configuration at Dockerfile in the root directory of the the Bolt project site. the project level is the recommended Bolt projects and assign it the follow- To use Bolt on Windows, you need approach. ing commands: Chocolatey [3], a package manager that performs typical functions such Creating a Bolt Directory from rastasheep/ubuntu-sshd as downloading and installing appli- run apt-get update && U cations. To install the Bolt packages, Your tasks in Bolt start with creating a and refresh the environment, run the Bolt project and setting up the targets. apt-get -y install libssl-dev commands: A Bolt project is a directory that in- expose 80 cludes the project-specific configura- cmd [\"/usr/sbin/sshd\", \"-d\"] choco install puppet-bolt tion settings. The first step is to create refreshenv the project directory and convert it The Dockerfile defines an Ubuntu into a Bolt project: container, including the SSH service, To import the Bolt PowerShell mod- which lets you talk to the entities in- ules, type mkdir first_bolt_project volved. The next step is to create the bolt project init first_bolt_project docker-compose.yaml file, which gener- Install-Module PuppetBolt ates two container instances. To do WWW.ADMIN-MAGAZINE.COM ADMIN 70 47

TOOLS Orchestration with Puppet Bolt this, create the file in your project di- To execute the whoami command on parameters: rectory and assign the settings shown target 1, for example, you would type: action: install in Listing 1. name: apache2 Now with the Compose file and the bolt command run whoami -t 127.0.0.1:2000 U help of the Dockerfile, create two con- -u root -p root --no-host-key-check description: \"Installs Apache\" tainers named target1 and target2. The SSH connections point to ports 2000 This command targets a system In the first section of the Bolt plan, and 2001, respectively, and the HTTP with IP address 127.0.0.1 and port you need to define the parameters that connections to ports 3000 and 3001. To 2000, through which SSH is address- your plan will accept. In this example, create and run the Docker containers able. The command also passes it is the TargetSpec type, which you and make sure the two containers are in a username and password to use to pass multiple targets to a plan. running, use the commands: enable access to the system. The For the parameters specified in the --no-host-key-check option disables plan to be used, you need to pass in docker-compose up -d --build certificate authentication. Typical out- the --targets option at the command docker-compose ps put for the command would be: line during execution. The steps section is where you spec- Screen output should appear tell- Started on 127.0.0.1:2000... ify the main part of your plan. In this ing you that the Docker containers Finished on 127.0.0.1:2000: example, it is named install_apache are ready to use, which means you and uses the Bolt package task to in- can now run commands against the STDOUT: stall Apache on the target systems. containers. root Bolt makes generous use of these pre- defined task configurations – in par- Commands Against Targets Successful on 1 target: 127.0.0.1:2000 ticular, reused tasks are defined there. Ran on 1 target in 0.3 sec The plan also defines the actions to Before you start executing extensive be performed (installation) plus the plans, you will want to run various The output indicates that you have name of the package. Having the in- commands against the target – not successfully run your first Bolt com- stall.yaml file in place in the plans least to familiarize yourself with Bolt mand on a target system. However, subdirectory is important. To run the specifics. The general syntax for com- you will typically want to address a plan in a container group, use: mand execution is: whole group and not just one system. To do this, Bolt uses an inventory file bolt plan run apache::install -t containers bolt command run <command> U that lets you group an arbitrary num- --targets <target name> <options> ber of systems. The plan is designated by two seg- ments separated by a double colon. Listing 1: docker-compose.yaml Deploying Bolt Plans The first segment specifies the name of the module in which the plan re- version: '3' Plans let you link commands, scripts, sides, and the second segment speci- services: and tasks, combining them to create fies the plan file, but without the file powerful workflows. Basically, you extension. In this example, the plan target1: can use Puppet’s own language or is located in the Apache modules build: . YAML for your plans. directory and is named install.yaml. ports: The next example installs an Apache The name of the plan is therefore - '3000:80' web server on the Docker targets cre- apache::install. - '2000:22' ated previously. It also takes care of You can watch your plan running at container_name: target1 starting the Apache services and up- the console. Typical output is shown loads a simple home page. To begin, in Listing 2. target2: you need some installation instruc- In a practical use case, it makes sense build: . tions for the Apache web server. In to use Bolt to install Apache on your ports: the apache subdirectory, create the target systems. The orchestration - '3001:80' plans directory and the install.yaml software offers a solution. To begin, - '2001:22' installation script: create a script file in which you store container_name: target2 the specific startup parameters for parameters: the Apache startup process. Drop this Listing 2: Output from a Running Plan targets: script (i.e., start_apache.sh in this type: TargetSpec example) in the files subdirectory Starting: plan apache::install below the Apache module directory. Starting: task package on target1, target2 steps: Next, add the following block to your Finished: task package with 0 failures in 18.00 sec - name: install_apache install.yaml file: Finished: plan apache::install in 20.00 sec task: package Plan completed successfully with no result targets: $targets 48 ADMIN 70 WWW.ADMIN-MAGAZINE.COM

Orchestration with Puppet Bolt TOOLS - name: start_apache Advanced Bolt Commercial GUI script: apache/start_apache.sh targets: $targets One key feature of Bolt is its mod- Bolt is managed entirely at the com- description: \"Start Apache Services\" ules, which bundle plans and tasks mand line, which is unlikely to faze most into typical workflows to facilitate administrators. If it does, the commercial The next time the plan is executed, integration. Another benefit is that Puppet Enterprise offers a convenient web the Apache install will also start, you can share the modules with interface. This license is also worth consid- basically clearing the way to access third parties; therefore, you can ering for large organizations that require the various Apache web servers. Note perform identical actions on ex- integrated governance, more flexibility, and that Bolt assigns port 3000 to the first ternal networks. However, if you team-oriented workflows. It also includes target, port 3001 to the second, and plan to use elements, note that the the ability to scale automation features and so on. In this example, the default functionality of the modules often monitoring – with and without agents. home page is 127.0.0.1:3000 for the depends on other modules. If you first target. install the module from the Bolt if the associated plan uses the ap- To store your HTML pages on the vari- console, Bolt automatically manages ply_prep function. ous Apache installations, create a sim- these dependencies for you. To do You can control the execution of the ple HTML file, name it index.html, and so, the module is added to the mod- plugins with entries in the Bolt con- store it in the /apache/files directory ule section of the project configura- figuration files. of your plan configuration. Before you tion file (bolt-project.yaml). can upload, you first need to create a In the next step, Bolt resolves the Conclusions scr parameter of the String type in the modules and their dependencies parameter configuration. The extended and generates a Puppet file. Do not Puppet Bolt is an excellent adminis- configuration then looks like this: modify this file. Finally, Bolt installs the modules and dependencies in the trative tool for orchestrating typical parameters: module directory (module-dir), which targets: you will find in the Bolt project direc- management tasks. Its strengths lie in type: TargetSpec tory (.modules). src: Plugins also simplify orchestration. its simplicity, flexibility, and ability to type: String They support dynamic loading and modification of information at Bolt do without agents. For administrators Now extend the steps section, adding runtime, which means that Bolt ac- the following block of code: tions can be controlled in a targeted who can do without the convenience way. You can use three different types - name: upload_homepage of plugins: of a web interface in favor of a pow- upload: $src Q Reference plugins are used to re- destination: /var/www/html/index.html erful environment, Bolt is an exciting targets: $targets trieve data from an external source description: \"Upload the Homepage\" and store the data in a static data tool. However, if the lack of a web in- object. After running the plan again, you Q Secret plugins are extensions that terface is an issue for you, check out should see the new homepage when provide encryption and decryption you access 127. 0.0.1:3000 (i.e., the facilities. the “Commercial GUI” box for fun- address of the Apache installation). Q Puppet library plugins are used when installing Puppet libraries damental details of the commercially licensed Bolt variant. Q Info [1] Puppet Bolt: [https://puppet.com/docs/ bolt/latest/bolt.html] [2] Homebrew: [https://brew.sh] [3] Chocolatey: [https://docs.chocolatey.org/en-us/] Q WWW.ADMIN-MAGAZINE.COM ADMIN 70 49

CONTAINERS AND VIRTUALIZATION Azure Automation Manage updates and configuration with Azure Automation Pass Go Microsoft Azure Automation provides a cloud-based service for handling automation tasks, managing updates for operating systems, and configuring Azure and non-Azure environments. We focus on VM update management and restarting VMs. By Thomas Drilling Azure Automation is not just about addition to many Azure- and Micro- Automatic by OS (Windows Auto- Lead Image © bagwold, 123RF.com automation tasks in and for Azure. soft-specific collectors. matic Updates)). However, this only It is a cloud-based service that pro- The third major area relates to orches- works for selected operating systems vides automation features for a wide trating and integrating automation (i.e., Windows Server 2008 R2 SP1, range of scenarios that can be roughly with other Azure services and third- 2012 R2 Datacenter, 2016 Datacenter, divided into three basic areas, all party products. On the integration and 2019 Datacenter). three of which share a number of front, unsurprisingly, many Azure ser- Essentially, Azure supports au- Azure Automation features, such as vices already interact with Azure Au- tomatic guest system patching, schedules, modules, credentials, and tomation. Even if you haven’t actively on-demand patch assessment, and certificates. dealt with the platform yet, you’ve on-demand patch installation only The first area covers repeatable and probably come into contact with the for VMs that you create from images consistent infrastructure provision- service indirectly once or twice – for that have the right publisher, offer- ing according to the infrastructure- example, when creating a virtual ma- ing, and stock keeping unit (SKU) as-code principle. Azure Resource chine in Azure. The Auto-shutdown combination within the list of theo- Manager Templates (ARMs), Azure feature in the Operations section of retically supported operating system Bicep, and Terraform are three popu- any Azure virtual machine (VM) is images. Unfortunately, this means lar technologies that can be used for also based on Azure Automation. In custom images or other publisher, this purpose. Another large sector this article, I look at further sample offering, and SKU combinations are revolves around event-based auto- applications. not supported. mation, such as for diagnostics and Additionally, the VM itself must meet problem resolution. Automatic Guest System a number of requirements for guest The second area is automated threat Patches system patches to work. For example, analysis, which can be performed the VM in question must have the in the context of incident detection In the management section of the pro- Azure VM Agent for Windows or in a security information and event visioning wizard for a new VM in the the Azure Linux Agent installed. management (SIEM) system – one Azure portal, users have – for some Furthermore, Windows VMs must example being Microsoft Sentinel, time now – been able to select a num- run the Windows Update service for which comes with well over 100 ber of options for patch orchestration Windows virtual machines. Of course, third-party system collectors in in the Guest OS updates section (e.g., the VM needs to be able to access the 50 ADMIN 70 WWW.ADMIN-MAGAZINE.COM