SATURN 2020

Prompted by the successful application of Artificial Intelligence (AI) and specifically Machine Learning (ML) in the commercial sector, the Department of Defense (DoD) seeks to leverage these technologies in military systems. AI offers the potential to solve a wide variety of DoD problems, increasing levels of autonomy and offering improved human-machine collaboration for our warfighters. However, the trustworthiness we place in such systems to function safely and ethically is a critical concern. The “black box” nature of ML combined with well-known sensitivities to the data sets used to develop (train) ML models raises many legitimate questions related to this trustworthiness question. The recent 2019 update to the National AI Research and Development Plan continues to acknowledge this gap.

We present a risk-based approach that considers the full system context and the full system life cycle, from initial concept through sustainment. While investment in research programs, such as DARPA’s Explainable AI (XAI) and Guaranteeing AI Robustness against Deception (GARD), is critical to achieving our long-term trustworthiness of AI goals, we believe it is also worth looking at the problem from a broader perspective. Our approach acknowledges (and leverages) the fact that AI algorithms are but one part of a complete system, and that decisions made as early as CONOPS and architecture development affect the system-level trustworthiness. Our approach focuses attention on the risks associated with AI algorithms embedded in a system producing incorrect results (e.g., classifications, decisions). To mitigate those risks, we use a collection of techniques applied to the system-level CONOPS and architecture, to validate and verify the AI algorithms and models embedded within the system, and to support a comprehensive test hierarchy supported with an automated DevOps pipeline. Our approach also leverages a collection of advanced techniques to mitigate the most significant AI trustworthiness risks.

Software is a growing component of modern business- and mission-critical systems. As organizations become more dependent on software, security-related risks to their missions are also increasing. Traditional cybersecurity approaches rely on addressing security risks during the operation and maintenance of software systems. However, the costs required to control security risks increase significantly when organizations wait until systems are deployed to address those risks. It is more cost effective to address software security risks as early in the lifecycle as possible.

Software programs should start managing cybersecurity risk early in the system's software lifecycle (e.g., during the requirements, architecture, and design phases). A complicating factor is that most software-intensive systems are networked and must operate within system-of-systems (SoS) environments. While networking offers many operational efficiencies to a system’s stakeholders, it also expands a system’s cyber-risk profile. Cyber attacks with the potential for mission impact can target any system within an SoS environment, creating complex attack vectors that must be considered during cyber-risk analysis. Software-intensive systems must be designed and architected with the knowledge that they must function as intended in an increasingly contested, challenging, and interconnected cyber environment.

For several years, researchers from the CERT Division of Carnegie Mellon University’s Software Engineering Institute (SEI) have been investigating how to enable mission success in SoS environments. The product of this research is the Security Engineering Risk Analysis (SERA) Method, a scenario-based approach for analyzing complex cybersecurity risks early in the system's lifecycle to support development of mission-critical software systems. The SERA Method incorporates a variety of models that can be analyzed at any point in the lifecycle to (1) identify security threats and vulnerabilities and (2) construct security risk scenarios. An organization can then use those scenarios to focus its limited resources on controlling the most significant security risks.

This presentation will describe the SERA Method and provide real-world examples of applying the method to analyze architectural and design weaknesses in complex weapon systems that operate in SoS environments. Attendees will learn how learn the basics of applying the SERA method to identify potential architectural weaknesses that attackers might be able to exploit.

Most of the time, machine learning (ML) is strongly viewed from a data science perspective. This means, you can find tons of information on algorithms and the treatment of data. However, what it actually means to architect systems, in which ML plays a role, is rather rarely found. Our focus is on the engineering of typically large systems, which are, to some degree, basing their functionality on machine learning. As such systems often serve in production large amounts of users, the fulfillment of quality attributes is critical and needs consideration in architecture design.

In this talk, we systematically decompose in the language of software architects what it means to build a system based on machine learning. We outline an architectural design space and discuss central architecture decisions an architect has to make when designing a system based on ML.

• This includes a perspective on both, the development time, and the runtime.
• We show how a system can be decomposed and how machine learning components look like and behave in the context of an overall system.
• Machine learning is fundamentally depending on data: ; Thus, the data aspect is central in our architectural considerations.
• As neural networks are very widespread nowadays for the realization of ML-based systems, we take a closer look at their architectural implications.
• We include a perspective on the activities around data collection, preparation, model selection and training, and model inference.
• We discuss deployment options for model training and model inference.
• We discuss different types of technologies available for machine learning, from as-a-service APIs over pre-trained models down to pure libraries requiring to construct and to the train the full model.

With this overview, architects will get the big picture of designing ML-based systems and have a much better position to bridge the gaps between data scientists, data engineers, and software developers and architects.

MongoDB, Cassandra, DynamoDB, MarkLogic, Couchbase, HBase, Redis, and many more NoSQL data stores; Hadoop and its vast ecosystem, Storm, Spark. These are some of the prominent platforms for big data storage and processing. Each of these platforms has its own unique strengths, architecture, and use cases for which they are a good fit.

In this session, we’ll look at the key architecturally-significant characteristics of some of the leading solutions and their categories, and discuss:

• key concepts
• essential architecture elements
• important characteristics in terms of performance, data volume, scalability, availability, etc.
• use cases where each platform is a good fit
• some prominent usage examples
• current state; the likely future direction

Today’s software engineer must not only write code – they also may have to build, test, secure, deploy and operate it as well. As a result, everyone in IT must undergo a personal transformation from a specialist to a T-shaped, multi-domain engineer. Enterprises are quickly recognizing that skills transformation is as important as tool or digital transformation.

Most practitioners can identify their deep competency (the stem of the T). But which broad skills should they be adding to the Top of their T and how do you know?

For the past two years, the DevOps Institute has fielded the Upskilling: Enterprise DevOps Skills Report based on a global community survey. This session will explore the year over year survey results including which process, functional, technical and soft skills are considered “must have”, “nice to have” and not as important. The session will also look at differences in results based on role, region and size of organization.

Industry and government alike have started to implement automated delivery pipelines only to have continued challenges with quality, validation, verification, and performance. A root cause is lack of requisite data provided to design, engineering, and delivery teams. Organizations need a deliberate strategy, process, and tool suite for the creation and management of SDLC support data.

DevOps and modern software delivery have a direct dependency on vigorous data profiling and subsequent synthetic data generation, self-service dataset reservation/refresh, as well as exposure to data integrations, often via an API. The move to data-centric and API-led architectures mandate clear understanding and access to datasets. Further, use of techniques like Service Virtualization to mock target data signatures and APIs needs to have appropriate process and "just enough" governance to deconflict data ownership across data stewards.

Historic approaches such as masking production data provides only a roughly equivalent volume of data and fails to provide the boarder cases needed for dependable development and testing.

Industry is just beginning to address through creation of new roles in the enterprise, such as a chief data officer, DataOps engineers, and data governance automation engineers, to provide holistic understanding of data and provide the necessary governance and rapid exposure of data sets across the full SDLC.

Attendees will engage to discuss current challenges and think through areas industry must address for test data under this very large DevOps/ continuous engineering umbrella.

• Identify types of data needed across the SDLC, likely by solution-type (architectural profile).
• Share lessons from implementation of SLDC data enablement (people, processes, and supporting technology).
• Explore data profiling, synthetic generation, and overall test data management (TDM).
• Explore governance challenges with data sets and the role of data in the DevOps/DevSecOps pipelines.
• Discuss evolving data-centric roles in government and the impact.
• Deliberate role of service virtualization and the nuances to adoption.

Participants will walk away understanding the need for data management and generation as well as techniques and tools used by my teams in the federal and state government domain. They will be armed with challenge areas including both technical as well as some in inherent people/cultural implications.

With the growing complexity of Computed Tomography Systems, big-bang integrations become an incalculable business risk, and with more than 100 developers world-wide DevOps caters to the need for risk reduction, shorter time-to-recovery, and faster time-to-market.

This talk presents a summary of our journey towards applying DevOps methods within medical device development. We defined our DevOps goal to mean deploying an in-house installation equivalent to a customer site. Our starting point was to enable continuous deployment and fully automate the chain from code-change to full system installation.

The software structure/architecture and the development processes and tools are subsequent improvement areas. We were missing efficient support for small change sets and the evolution into a monolith prevent us using small binary-artifact based components for integration. We describe the reasons and impact, and present first results from improvements.

Further we lay out the challenges regarding the necessary cultural change, which has additional aspects within system development compared to software-only development. We explain not only the success, but also the setbacks that we encountered.

The talks closes with an outlook on concrete next steps but will also shed some light on visionary aspects like "continuous compliance" to help enable direct deployment to customer sites.

During this workshop we will give a crash course in threat modeling. Threat modeling is a structured activity for identifying and evaluating application threats and design flaws. You use the identified flaws to adapt your design, and scope your security testing. Threat modeling allows you to consider, identify, and discuss the security implications of user stories in a structured fashion, and in the context of their planned operational environment.

This threat modeling workshop will teach you to perform threat modeling through a series of exercises, where our trainer will guide you through the different stages of a practical threat model based on an Amazon Web Services (AWS) and microservices migration from a classical web application. At the end of the workshop we will create a complete threat model of a CI/CD pipeline.

This workshop is meant for security champions, application architects, developers and security people. Basic understanding of software development, microservice architecture and cloud platforms (AWS) is recommended. No previous threat model knowledge is needed.

As a software architect you improve the overall security posture of a system you are designing with threat modeling:
-Threat modeling and architecture both use models of systems and involves analyzing (security) properties of those models.
-Threat modeling also leads to an understanding of architectural choices with security implications, gives structure to a set of security challenges, and enables you to change the architecture in a way that reduces these problems at a manageable cost.

This workshop is 50% hands-on, we will challenge the attendees to go through different exercises, built upon a fictional Acme Hotel Booking (AHB) system:
-diagram the AHB applications, sharing the same REST backend (15 minutes)
-threat identification, migrating the AHB applications to AWS. (15 minutes)
-AHB threat mitigation of microservices and S3 buckets (15 minutes)
-threats and mitigations for the AHB CI/CD pipeline (10 minutes)

In this tutorial we first demonstrate how to fully automate architecture analyses using data collected from software development processes. These analyses are supported by the DV8 tool suites: measuring and tracking architecture maintainability, pinpointing architecture anti-patterns, quantifying architecture debts, and analyzing the potential return on investment (ROI) of refactoring. We will present several case studies showing how these analyses have worked in practice, and provide hands-on training guiding the audience through the process step by step, from collecting project data, generating analysis results, interpreting architecture health reports, understanding anti-patterns, ROI data and debts, and finally making informed refactoring decisions.

If a project recognizes the need to monitor its architecture health, then this analysis must be integrated into its development process. We will also demonstrate how DV8 can be integrated into CI process and SonarQube, so that they can be executed at any time: following a commit, a sprint, on a nightly build, or on demand. Project data is crucial: (1) it is objective and the ground truth of a system, and (2) it reflects the quality of the development process. We will also demonstrate how to assess process data quality in this tutorial.

Microservices set high standards for architecture and infrastructure: asynchronous message-based applications that are automatically deployed, containerized, scaled, and managed. This session focuses on microservices architecture, microservices design patterns, anti-patterns and some strategies to convert a monolith to microservices. The talk provides answers to the following questions: What are the differences that people might not be familiar with between a monolith and microservices? What do most people or businesses get wrong about microservices? What kind of tools do you use when managing microservices? And finally, what are the drawbacks of implementing microservices and how to deal with them? After this talk, you will have learned the full spectrum of deploying a microservices architecture.

Applications composed of microservices increase agility, as well as, embrace elasticity. Loosely coupled through language-agnostic APIs, microservices can be built on the runtime best suited to the workload. However, this flexibility increases complexity. Platforms reduce this complexity by providing a composable management backplane and common domain services. This session will introduce a lean platform design approach to rapidly deliver microservice platforms while mitigating bloat and application development bottleneck risks.

More and more teams capture their design decisions in Architectural Decision Records (ADRs). These ADRs are commonly used to document decisions after they have been taken. We will share a visual format for ADRs which has proven to also be useful in the period leading up to the decision itself, in which a team learns about a decision's context and business aspects, finds alternatives, and finally makes the trade-off between them. We call this format 'decision stories.' We developed and optimized the format by applying several iterations in practice over the past few years.

We will share our experiences, which show that the format not only captures a decision and its rationale, but also helps guide the decision process and engage stakeholders. The format is intuitive in use, and has shown to improve communication, both within teams as well as with external stakeholders.

In this experience report, you will hear how we developed our design decision story template, and learn how to get the most out of the format in practice.