What is augmented reality?

Anything we can do to improve our knowledge of our immediate physical, cognitive, social, and informational context, our personal state, threat awareness, and opportunity awareness, without unduly complicating matters — Clark Dodsworth.

Clark Dodsworth

Clark Dodsworth

Clark Dodsworth is a product strategist for next wave smart devices. According to Clark, any product is a tool, any tool is an interface: whether it’s software or hardware,… whether it involves interaction design, industrial design, visual design, public-space design, design management, or all of them. A tool should fit the user. A tool should adapt to the user.

Clark works with companies to improve tools, evolve tools, add and revise features of them, position them in the marketplace, and find the sweet spot for IP to become new tools.

Context is king

In 2010, Clark presented at the Augmented Reality Event in Santa Clara, CA. His topic was Context is King — AR, Salience, and the Constant Next Scenario. Highlights from this talk follow.

Eight layers of context awareness

Eight layers of context awareness

Context is the interrelated conditions in which something exists or occurs (Merriam Webster). It is the situation within which something exists or happens and that can help explain it (Cambridge Dictionary). It is any information that can be used to characterize the situation of an entity (Dey, 1999). It is the set of environmental states and settings that either determines an application’s behavior or in which and application event occurs and is interesting to the user (Chen, Kotz 2000).

This diagram (above) depicts eight categories of context for smart services. Context-aware applications look at the who, where, what, and when of entities to determine why and how a situation is occurring.

What happens when there’s a software layer that learns you?

Augmented reality to cope with your constant next scenario

Augmented reality to cope with your constant next scenario

Augmented reality (AR) on smartphones, tablets and other platforms is a new and highly malleable set of tools and substrates for them. It is the path to how we can manage our daily work and life in the midst of the tsunami of of information we can get all the time, not to mention the daily social contacts we manage and deal with. The next steps for AR are about helping each user with the constant next scenario in their day, minute by minute…anticipating and supporting their process, without interrupting it. While we’ll get there step-by-step, we need to have this goal in mind, or we will undershoot the opportunity and delay very real efficiencies in our daily lives and work.

The preceding diagram is a data flow concept model of the information you generate as you proceed through your day, as well as the informational data you normally receive (text, voice, email, alerts, etc.), and how those streams should be processed by the three segments of hypersonalization services (App, OS, and Cloud).

Also, it shows the step of selecting the available sensory modes for delivery, which includes dynamically formatting the information for display in whichever mode is most appropriate.

It also indicates that device sensors will tend to be in constant contact with either cloud-based sensor datastreams or peer-to-peer ac-hoc (mesh) sensor nets, or both, to adequately inform you of relevant changes and states of your physical environment. They will not only receive but also contribute to both, in part as a way to pay for access to the aggregate sensor data feeds.

Once these three methods of analytics (salience, live context, and the actual decision-path, have learned enough about you, they begin to do probabilistic data staging and delivery for your next decision(s).

Augmented Context + Information Salience = Value.  

Constantly supporting the next scenario of my daily life

Constantly supporting the next scenario of my daily life

Augmented reality means dynamic, context-driven hyperpersonalization, salience filtering, & scenario modeling in order to adaptively deliver info, highly personalized via N-dimensions of user states, with high space/time precision, and without spiking your cognitive load. There is constant auto discovery of data feeds and sensor feeds. There is constant markerless 2D & 3D feature identification and object recognition linked to motion analysis and evaluation. There is indoor position mapping  seamlessly linked with outdoor GPS position sensing.  There is nuanced gestural and integration with voice as well as  constant audio + visual awareness.

Allostatic control delivers only the data needed, when needed. “Allostasis” is maintenance of control through constant change. This means avoiding user distraction by constantly contextually evaluating what should be displayed to the user, if anything. Without sophisticated software “awareness” to drive what information is inserted into the user’s awareness,  we quickly hit the complexity wall. Providing the right amount of information we need, when we need it, and only the amount we need, in the sensory mode that’s most appropriate at the moment, is the doable goal.

Smart tools fit the user and the task. They adapt to — i.e. learn — the user over time, becoming more useful to the individual the more they’re used. Good augmented reality tools adapt dynamically. Otherwise ongoing management of their services is too distracting. Smart tools must provide their value to us without the amount of user configuring we’ve become accustomed to.

When systems record, analyze, and thus learn from user choices over time, they learn what scenarios you select at each moment. They can then begin to provide information services for the next impending choice situation – a set of scenarios you’ll choose from that may need information-services support. Every day.

The Context War

The Context War

Information vendors and access providers can create new revenue streams by becoming  trusted, high-security vendors of tracking and analysis of user’s paths through location, time, people, tasks, decisions, interests, alerts, ads, priorities, and the publicly shared priorities of his/her friends and associates. Providing a set of analytical services that can be selected and turned on and off by the user, and whose value accretes over time, will become viewed as the most effective way for individuals to parse the information they need to use in their day-to-day work and life. This is closely related to just-in-time learning, and constant on-the-job training, and life-long learning.

Smart systems and devices, context-aware computing, and augmented reality define the next battle field for managed device platforms. It’s all about long-term, sustaining, contextually dynamic, hyper-personalized relationships between users and networked delivered services. As shown in the following diagram, five vendor camps have formed along lines of research and are competing to create the dominant smart platform ecosystem. Contending camps include: Google (Android), Apple (iOS), Facebook, Amazon, and Microsoft.


Thanks Clark.

What is the addressable market for 
enterprise-situated knowledge workers?

Potential exists to increase knowledge worker productivity 50X by 2030.

Market for knowledge computing solutions

Market for knowledge computing solutions

This table provides a conservative, order of magnitude estimate of the addressable market (number of individuals, organizations) for five categories of knowledge computing solutions. Markets for smart mobile internet of services and things will be orders of magnitude larger.

What are some principles for multi-agent systems?

Environment is an active process. A flock is not a big bird. Emergent behavior is distributed. Think small.

Hieronymus Bosch

Hieronymus Bosch

Software technologies have evolved from assemblers, to procedure programming, to object-oriented software. Goal-oriented direct model-driven services, semantic agents, and multi-agent systems are the next software paradigm.

Some principles for multi-agent systems include:

  1. Agents should correspond to “things” in a problem domain rather than to abstract functions
  2. Agents should be small in mass, time (able to forget), and scope (avoid global knowledge action)
  3. Multi-agent systems should be decentralized (with no single point of control/failure)
  4. Agents should be neither homogeneous nor incompatible but diverse
  5. Agent communities should include a dissipative mechanism (entropy leak)
  6. Agents should have ways of caching and sharing what they learn about their environment
  7. Agents should plan and execute concurrently rather than sequentially.

What is goal-oriented software engineering?

Declaring knowledge about data, processes, rules, services and goals separately from application code to enable sharable, adaptive, autonomic, and autonomous solutions.

Almost Human

Almost Human

Goal-oriented software engineering separates knowledge (i.e., semantic models) about what to do from how to do it. Goals identify targets for a process, or portion of a process. Plans get selected dynamically to achieve goals. Self management within the business process logic allows dynamic restructuring of goals and plans at run time as well as adaptation of process behavior to prevailing run-time conditions and business contexts.

Autonomous solutions have the ability to independently perform some processing toward achieving or maintaining one or more goal. They are self-contained systems that function independently of other components or systems by acquiring, processing and acting on environmental information.

Autonomic computing systems have the ability to self-regulate a goal/plan hierarchy to make automatic adjustments in real-time in accordance with changing environment using closed control loops without the need for any direct intervention.

Alan Kay

Alan Kay

Alan Kay

“Don’t worry about what anybody else is going to do… The best way to predict the future is to invent it. Really smart people with reasonable funding can do just about anything that doesn’t violate too many of Newton’s Laws!”
— Alan Kay

What makes concept computing different?

Webs of meanings and knowledge. Constraint-based, direct model execution systems that know and can share what they know. Architectures of learning that perform better with use and scale.

Three stages of semantic solution envisioning

Three stages of semantic solution envisioning

The diagram above depicts three stages of semantic solution envisioning. Business capability exploration focuses on reaching agreement about goals, constraints, and the basic concepts and terms that different groups use. Different stakeholders express their requirements in their own language. Solution exploration defines needed capabilities and concept computing solution patterns. Semantic software design is knowledge-centric, iterative and incremental. Changes in ontology, data, rules, workflow, user experience, etc, can automatically update system functionality, without impacting underlying legacy systems and databases.

The process of semantic solution development is declarative, direct execution model-driven, and knowledge-centric and rather than being procedural, specification waterfall translation-driven, and document centric. Build cycles are fast, iterative, non-invasive. Semantic solution development typically entails less time, cost, and risk to deploy, maintain, and upgrade.

Knowledge is extracted and modeled separately from documents, schemas, or program code so it can be managed integrally across these different forms of expression, shared easily between applications (knowledge-as-a-service), aligned and harmonized across boundaries, and evolved. Requirements, policies, and solution patterns expressed as semantic models that execute directly can be updated with new knowledge as the application runs.

Semantic solutions emerge from a techno-social collaboration that also supports do-it-yourself development. The development process is business and user driven versus IT and developer driven. The collective knowledge developed is interpretable by different human stakeholder as well as machine executable. Different skills are key including domain expertise, concept computing architecture, knowledge modeling, and (smart) user experience design.

How is software’s space vs. time trade-off evolving?

Data-centric, constraint-based declarative processes trump procedural algorithms.

Complex reasoning at scale demands declarative processing

Complex reasoning at scale demands declarative processing

In software, we can trade processing cycles (time) for storage (space). For example, imagine an algorithm where every possible input combination, processing sequence, and output variation had been pre-computed and that the results of all these possible states had been stored. Next time there would be no need to (re)execute the program because all possible inputs and results have already been computed and declared. You simply look up the result by specifying the variables that constrain the space of possible outcomes. Nothing could be faster.

Historically, the cost of computation favored executing algorithms over and over, rather than providing the semantic bandwidth and memory to access and store all possible permutations in a declarative knowledge mesh. That was then.

Today, computing economics have reversed. Massive memory and parallel processing is cheap and getting cheaper. Massive declarative knowledge-based computing is now economical.

But, there’s an even more compelling reason why the shift is happening now — new capabiities.
Declarative systems allow applications that can learn, evolve, and use knowledge in ways the system designer could not foresee.

How is this possible?

  • Declarative languages specify “what” is to be done. Declarative representations are like a glass box with knowledge visible in a format that can be accessed, manipulated, decomposed, analyzed, and added to by independent reasoners.
  • Imperative (procedural) languages, on the other hand, encode “how” to achieve a particular result, step by step. Procedural representations are like a black box with knowledge fixed and locked inside an algorithm. Procedural systems do the same thing every time and cannot learn from experience. Making changes requires rebuilding data stores and rewriting algorithms off-line.

What is scale invariance?

The ability of a system to scale-down, scale-up, scale-out, and scale-in.

Four dimensions of scalability

Four dimensions of scalability

Scalability is the ability of a system, network, or process to handle a growing amount of work in a capable manner, and its ability to be further empowered to accommodate that growth. For example to handle more users, more data and sources, greater knowledge intensivity of decision making, increasing process dynamics, infrastructure expansion, and system adaptivity.

Architectures for the next stage of the internet will evolve along four key dimensions of scalability:

As Richard Feynman said, “There is plenty of room at the bottom!” Scale-down architecture is about maximizing feature density and performance, and minimizing power consumption. Approaches include nano-technology, atomic transistors, atomic storage, and quantum computing.

Scale-up architecture (or vertical scaling) is about maximizing device capacity. It adds more processing power, memory, storage, and bandwidth on top of a centralized computing architecture. Typically, scaling up uses more expensive, platform-specific, symmetric multiprocessing hardware. It’s like a fork lift to increase capacity and performance.

Scale out architecture (or horizontal scaling) is about maximizing system capacity. It adds multiple instances of the process architecture enabling the system to address vastly more subjects, services, things. Processors perform essentially the same task but address different portions of the total workload, typically using commodity Intel/AMD hardware and platform independent open source software.

Scale in architecture is about maximizing system density and minimizing end-to-end latency. Scale-in architecture differs from the old compute-centric model, where data lives on the disk in a deep storage hierarchy and gets moved to and from the CPU as needed. The new scale-in architecture is data-centric. Multi-threading within and across muti-core processors enables massive parallelism. Data lives in persistent memory. Many CPUs surround and use in-memory data in a shallow flat storage hierarchy where each memory location is addressable by every process thread.

Scale-in architecture is data-centric

Scale-in architecture is data-centric

Scale-in architecture empowers concept computing — a dimension of increasing knowledge intensivity of process, decision-making and user experience. From a big data perspective, it allows addition of new data elements, aggregation of context, and on the fly modification of schemas without the necessity of off-line rebuilds such as are needed to modify data warehouses and NOSQL processes at scale. From a business perspective scaling-in enables “systems that know” where changes to business requirements, policies, laws, logic etc can be managed flexibly and separately from data sources and operations. Scale-in architecture is a prerequisite for cognitive computing and systems that can learn and improve performance with use and scale.

These four dimensions of scalability are key to building smart, high-performing processes of any size, any complexity, any level of knowledge intensivity, and which can learn and grow live to quickly adapt to changing requirements and implement scale-free (or scale-invariant) business models.

Transformational change demands architecture that is different.

Transformation demands different architecture.

Transformation demands different architecture.

Here is a little story to illustrate the point:

A scientist discovered a way to grow the size of a flea by several orders of magnitude. She was terribly excited. After all, a flea can jump vertically more than 30 times its body size. She reasoned that a flea this big would be able leap over a tall building. Perhaps, there could be a Nobel Prize in this.

When the day came to show the world, she pushed the button and sure enough out came this giant flea, over two meters high. But, rather than leaping a tall building, it took one look around and promptly fell over dead. Turns out it couldn’t breath. No lungs. Passive air holes that worked fine for oxygen exchange in a tiny flea were useless for a creature so big.

What are lean services?

By rethinking and streamlining service processes, most companies can cut expenses by 10 to 30 percent and sharply improve the satisfaction of internal and external customers.

BCG's Lean Services Primer

BCG’s Lean Services Primer

One way businesses can improve their competitiveness is to apply lean manufacturing principles to services. In this report, Boston Consulting Group identifies six success factors for implementing lean services:

  • Identify and map key processes
  • Reduce complexity
  • Standardize work modules
  • Harness the power of “big data”
  • Set and track performance metrics
  • Cross-train to increase productivity.

In health care, banking, travel, and other service industries, the customer is the product moving through the process — and experiencing firsthand the frustration of inefficiency. Because processes typically cross functions and departments, few people involved with them have a complete picture of the end-to-end-workflow.

Key reasons for inefficient service processes include a lack of standardization and consistency, exceptions and rework that slow throughput, and an inability to analyze and manage the drivers of work force productivity and customer satisfaction.

Interdependencies and interfaces are often hidden. Inconsistency is a problem for many service processes. Different personnel have different methods and levels of skill. Often a small percentage of work eats up a disproportionate amount of time.

Standardizing best practices, cross-training, and refactoring the 80/20 distribution of work can lead to significant efficiency gains. Look for handoffs and steps that waste time or add no value. Analyze information flows to identify silos and roadblocks. Flag and eliminate any variations, disruptions, rework, or exceptions that slow the workflow. Harness the power of “big data” to gain insights to minimize waste, lower costs, and sharply improve service process performance.