NATURAL LANGUAGE INTERACTION WITH THE WEB OF
THINGS
DeborahA.Dahl
ConversationalTechnologies
MobileVoice2015
April20-21
Sa...
Connected things will be everywhere
• Everyday life – home appliances, entertainment, fitness
monitor, toys, pets…
• Indus...
Current interaction model
• Vendor provides an SDK for user devices
• Vendor provides an API for accessing the functions o...
Problem 1: UI’s
One Approach to UI’s: the basket of
remotes
Hundreds of
apps!
Another approach: Apps
UI Proliferation
user
Control/Interface
Mobile device
Wearable
Ambient device
(e.g. Amazon Echo/Jibo)
Apps
Apps
Apps
Users need a uniform way to interact
• Don’t burden the user with too many interfaces
• No matter how well-designed a GUI ...
Problem 2: Too many API’s
• Many different API’s for different things
• Want to hide details of each thing API to simplify...
The Answer: Standards!
• For natural language UI’s: W3C EMMA
• For API’s: The W3C Multimodal Architecture
EMMA: a standard format for natural
language inputs
user
Mobile device
Wearable
Ambient device
(e.g. Amazon Echo/Jibo)
Nat...
What’s in an EMMA document?
• Standard metadata about utterance
• Confidence, input tokens, alternative interpretations, p...
What about API’s?
• W3C Multimodal Architecture is generic API for
interacting with encapsulated functionality (Modality
C...
MMI generic modality component
Echonet Lite KNX Zigbee
Mobile
device
Wearable
Ambient
device
Command mapper
MMI Interactio...
Example: Light bulb
• Philips Hue Light Bulb
• LED light that can change
colors
• Controlled through LAN via
bridge
• Brid...
Standards-based
Control of Hue Light
<emma:emma
xmlns:emma="http://www.w3.org/2003/04/emma"
xmlns:xsi="http://www.w3.org/2...
How to Scale the Web of Things
• Too many UI’s?
• Use natural language
• Use EMMA for a regular format for natural languag...
of 17

Natural language interaction with the web of things

My presentation from the 2015 Mobile Voice Conference on "Natural language interaction with the web of things". I talked about how interacting with the web of things will become more and more difficult as the number of connected things increases. We need to have generic UI's based on natural language and generic API's based on multimodal standards to make life manageable for users as well as developers. You can also see a longer version of the light bulb demo at https://www.youtube.com/watch?v=7J0FPX524lA. The full set of Mobile Voice presentations is here http://avios.org/?page_id=5403.
Published on: Mar 3, 2016
Published in: Technology      
Source: www.slideshare.net


Transcripts - Natural language interaction with the web of things

  • 1. NATURAL LANGUAGE INTERACTION WITH THE WEB OF THINGS DeborahA.Dahl ConversationalTechnologies MobileVoice2015 April20-21 SanJose
  • 2. Connected things will be everywhere • Everyday life – home appliances, entertainment, fitness monitor, toys, pets… • Industry – office, building, industrial robot, parts for manufacturing… • Public environments – retail store, city services, highway, museum… • Medicine – sensors, alarms… • Transportation -- car, train, bus, airplane
  • 3. Current interaction model • Vendor provides an SDK for user devices • Vendor provides an API for accessing the functions of their connected things • Company and/or Developers create apps for various user devices using provided API’s • Users buy connected things and use GUI apps to interact with things What’s wrong with this approach?
  • 4. Problem 1: UI’s
  • 5. One Approach to UI’s: the basket of remotes
  • 6. Hundreds of apps! Another approach: Apps
  • 7. UI Proliferation user Control/Interface Mobile device Wearable Ambient device (e.g. Amazon Echo/Jibo) Apps Apps Apps
  • 8. Users need a uniform way to interact • Don’t burden the user with too many interfaces • No matter how well-designed a GUI interface is, it’s different from other GUI’s • User has to learn it • Harder for users like older adults or people with cognitive disabilities • UI’s can be aggregated into categories (like one UI for the connected home) • Natural language provides a uniform UI across connected things
  • 9. Problem 2: Too many API’s • Many different API’s for different things • Want to hide details of each thing API to simplify development
  • 10. The Answer: Standards! • For natural language UI’s: W3C EMMA • For API’s: The W3C Multimodal Architecture
  • 11. EMMA: a standard format for natural language inputs user Mobile device Wearable Ambient device (e.g. Amazon Echo/Jibo) Natural language Understanding + graphical input EMMA
  • 12. What’s in an EMMA document? • Standard metadata about utterance • Confidence, input tokens, alternative interpretations, process, timing… • Interpretation of utterance in a structured format • Key-value pairs, for example • Groupings with related utterances • A standard way to represent natural user inputs
  • 13. What about API’s? • W3C Multimodal Architecture is generic API for interacting with encapsulated functionality (Modality Components) • Life-cycle events like “start”, “cancel”, “pause”, “resume” • Coordinated by an Interaction Manager • Work on dynamic systems is starting
  • 14. MMI generic modality component Echonet Lite KNX Zigbee Mobile device Wearable Ambient device Command mapper MMI Interaction Manager Speech Recognitio n Natural language understandin g User devices Interpretation Modality Components EMMA over MMI EMMA Over MMI MMI Life-cycle events things W3C Standards Other standards Putting things together Car Audio over HTTP or Web Sockets
  • 15. Example: Light bulb • Philips Hue Light Bulb • LED light that can change colors • Controlled through LAN via bridge • Bridge communicates to light bulbs via Zigbee protocol • Has own API for developers • 45 apps on Google Play, 70 on iTunes Store, 3 Windows available
  • 16. Standards-based Control of Hue Light <emma:emma xmlns:emma="http://www.w3.org/2003/04/emma" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.1" xsi:schemaLocation="http://www.w3.org/2003/04/emma http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"> <emma:interpretation emma:confidence="0.744" emma:process="wit.ai" emma:tokens="it's dark in here" id="interp237 <nlResult> <_text>it's dark in here</_text> <msg_id>7e5e96f7-4a4d-4f62-9449-43c455a530f7</msg_id> <outcomes> <e> <_text>it's dark in here</_text> <confidence>0.744</confidence> <entities> <on_off> <e> <value>on</value> </e> </on_off> <thingType> <e> <value>light</value> </e> </thingType> </entities> <intent>changeState</intent> </e> </outcomes> </nlResult> </emma:interpretation> </emma:emma>
  • 17. How to Scale the Web of Things • Too many UI’s? • Use natural language • Use EMMA for a regular format for natural language results • Too many API’s? • Use MMI Architecture to encapsulate things as Modality Components

Related Documents