Untangling Enterprise Software Development

Information, Discovery, Analysis, and Artificer

Brett Meyer

  • Artificer, Project Lead
  • Senior Software Engineer, Red Hat
  • @brettemeyer

Typical enterprise software development...

The Problem

  • Companies, teams, individuals
  • Tangled mess of "stuff"
  • Information, metadata, and "artifacts"

More Problems

  • Re-use and prevent duplicated efforts: hard
  • Understand how everything connected, related, dependent: really hard
  • Minimal "breaking things" (early detection): extremely hard

Even Worse

  • Same problems, but within granular content of artifacts
  • Ex: XSD type declaration <-> WSDL message

Oh So Many Problems

  • Tends to include a wide variety of bits
  • Files, including granular content
  • Files, no granular content considered
  • Custom metadata for each file
  • Logical, metadata-only "artifacts" (information)

Your Frustrations

(Hoping someone actually understands that reference...)

What's Needed?

Discovery & Re-Use

  • Has this file/capability/project been created before?
  • What pieces could I reuse?
  • Who's responsible for this piece?
  • How do I even find the pieces when everything is so desperate?

Impact & Dependency Analysis

  • Who uses this artifact?
  • How does this artifact fit into the larger picture?
  • If I change this file, who's affected?
  • If I change this specific element in a file, who's affected?

"Bigger Picture" Analysis

  • How does this artifact fit into our projects?
  • How does this artifact fit into our organization?

Hierarchical Classification

  • "Tags", but more to it...
  • Hierarchical ontologies

Lifecycle Management

  • Thin interfaces only!
  • Workflow utility

Auditing

  • Show me how this artifact has changed over time
  • Show me who all changed this artifact

What has worked for your organizations?

Artificer

Disclaimer

  • No one likes sales pitches...so it won't be.
  • General discussion of approaches, using Artificer as an example
  • But it is pretty sweet...
  • This is a discussion and fact finding mission!

Artificer Overview

Common Data Model

  • Desperate information
  • Extremely varied
  • Vital: normalizing into common data model
    • Abstract and flexible
    • Powerful and all-encompassing, OOTB
    • Extendible

Common Data Model (cont'd)

  • Physical and logical artifacts
  • Interconnected (bi-directional)
  • Hierarchical

Common Data Model (cont'd)

  • "Modeled" metadata: provide the "core" data OOTB
  • Auto-detection of artifact types
    • Not just MIME!
    • "If file 'foo' is present in the archive, it's A. Otherwise, B."
    • "If the content includes 'foo', it's A. Otherwise, B."
  • Ad-hoc metadata

Common Data Model (cont'd)

  • "Primary": main artifact
  • "Derived": logical artifacts pulled from primary content (ex: type declarations from XSD)

Derivation

  • Auto creates "derived" logical artifacts from the primary
  • Examples:
    • Type declarations in XSDs
    • Messages/ports/endpoints in WSDLs
    • Many more...

OOTB Data Types

  • Provide "enough" OOTB to be immediately useful
  • Examples:
    • Basic "document"
    • XML
    • XSD
    • WSDL
    • WS-Policy
    • Java
    • Office
    • (Artificer also provides plugins for other types of JBoss projects)
    • (Many more planned...)

Logical Data Types (Examples)

  • Service information
  • Teams
  • Metrics
  • Policies
  • Deployment information

Relationships

  • Bi-directional
  • Between any artifacts
  • Relationship name
  • 1..n targets

Relationships (cont'd)

  • Maybe the most important piece to get right
  • Relationships between both primary and derived
  • Support both automatic relationships and ad-hoc
  • Auto relationship examples: WSDL imports XSD, WS doc'd by WSDL, etc.

Classifications

  • Hierarchical "tags"
  • "Hierarchical" == powerful, especially for queries
  • Web Ontology Language (OWL)

Query Language

  • Encapsulates all areas of metadata
  • Can chain 0..n relationship paths
  • Full text search

Smart Constraints

  • Block certain actions
  • Much like SQL FK constraints
  • Preventing impacts > impact analysis

Extensibility

  • Compile-type contracts
    • Artifact "builders": derivation, custom metadata, etc.
    • Artifact type detection
  • Ad-hoc everything

Interfaces & Integrations

  • REST/ATOM
  • Web UI
  • SDKs
  • CLI
  • Others: EJB, JMS, etc.

Visualizations

  • Really hard
  • Useful for impact analysis and dependencies overview
  • Visualize the relationship graph
  • Need to be "smart" and show enough, but not too much

Artificer Specifics

  • Everything we just talked about!
  • "Not Just Another Repo"
  • Powerful data, powerful tooling
  • Can be nearly "auto everything"
  • Valuable info, relationships, context, etc. over simple repos

Artificer Specifics (cont'd)

  • Maven (& Gradle): Includes a Maven Repo "Facade"
    • Acts just like any other Maven repo
    • Publish and consume
    • Can replace your existing repo, or augment it

Artificer Specifics (cont'd)

  • Performant, vertically scalable, horizontally scalable
    • RDBMS + JPA (Hibernate) + Caching (Infinispan) + Indexed Search (Lucene/Hibernate Search)
    • JCR (ModeShape) + Infinispan
    • Planned: NoSQL + Hibernate OGM
    • Maybe: Graph DBs

Artificer Specifics (cont'd)

  • Mostly built on solid JBoss platforms:
    • Wildfly/EAP
    • Hibernate Family
    • ModeShape
    • Keycloak
    • Errai
    • Aesh

Artificer Specifics (cont'd)

  • OASIS S-RAMP
    • Fully-compliant with OASIS S-RAMP spec
    • ...for now (fairly dead spec)
    • Foundation of data model, query syntax, and REST/Atom binding
    • Mostly SOA-centric
    • We extend well beyond the spec

Artificer Specifics (cont'd)

  • Docker (and therefore OpenShift V3)
  • IDE (Eclipse) integration, coming soon
  • Multiple extension point contracts, but...
  • ...soon replacing with a new, powerful plugin architecture

Artificer Features & Ideas

Out of Left Field

  • Software was the original purpose
  • Useful in many other ways
  • Any type of hierarchical, inter-connected information

Example: "Getting Things Done" (David Allen)