Tag Archives: Watson

AI Personal Assistant Needs Remedial Guidance for their Users

Providing Intelligent ‘Code’ Completion

At this stage in the application platform growth and maturity of the AI Personal Assistant, there are many commands and options that common users cannot formulate due to a lack of knowledge and experience.  Using Natural Language to formulate questions has gotten better over the years, but assistance / guidance formulating the requests would maximize intent / goal accuracy.

A key usability feature for many integrated development environments (IDE) are their capability to use “Intelligent Code Completion” to guide their programmers to produce correct, functional syntax. This feature also enables the programmer to be unburdened by the need to look up syntax for each command reference, saving significant time.  As the usage of the AI Personal Assistant grows, and their capabilities along with it, the amount of commands and their parameters required to use the AI Personal Assistant will also increase.

AI Leveraging Intelligent Command Completion

For each command parameter [level\tree], a drop down list may appear giving users a set of options to select for the next parameter. A delimiter such as a period(.) indicates to the AI Parser another set of command options must be presented to the person entering the command. These options are typically in the form of drop down lists concatenated to the right of the formulated commands.  Vocally, parent / child commands and parameters may be supplied in a similar fashion.

AI Personal Assistant Language Syntax

Adding another AI parser on top of the existing syntax parser may allow commands like these to be executed:

  • Abstraction (e.g. no application specified)
    • Order.Food.Focacceria.List123
    • Order.Food.FavoriteItalianRestaurant.FavoriteLunchSpecial
  • Application Parser
    • Seamless.Order.Food.Focacceria.Large Pizza

These AI command examples uses a hierarchy of commands and parameters to perform the function. One of the above commands leverages one of my contacts, and a ‘List123’ object.  The ‘List123’ parameter may be a ‘note’ on my Smartphone that contains a list of food we would like to order. The command may place the order either through my contact’s email address, fax number, or calling the business main number and using AI Text to Speech functionality.

All personal data, such as Favorite Italian Restaurant,  and Favorite Lunch Special could be placed in the AI Personal Assistant ‘Settings’.  A group of settings may be listed as Key-Value pairs,  that may be considered short hand for conversations involving the AI Assistant.

A majority of users are most likely unsure of many of the options available within the AI Personal assistant command structure. Intelligent command [code] completion empowers users with visibility into the available commands, and parameters.

For those without a programming background, Intelligent “Command” Completion is slightly similar to the autocomplete in Google’s Search text box, predicting possible choices as the user types. In the case of the guidance provided by an AI Personal Assistant the user is guided to their desired command; however, the Google autocomplete requires some level or sense of the end result command. Intelligent code completion typically displays all possible commands in a drop down list next to the constructor period (.). In this case the user may have no knowledge of the next parameter without the drop down choice list.  An addition feature enables the AI Personal Assistant to hover over one of the commands\parameters to show a brief ‘help text’ popup.

Note, Microsoft’s Cortana AI assistant provides a text box in addition to speech input.  Adding another syntax parser could be allowed and enabled through the existing User Interface.  However, Siri seems to only have voice recognition input, and no text input.

Is Siri handling the iOS ‘Global Search’ requests ‘behind the scenes’?  If so, the textual parsing, i.e. the period(.) separator would work. Siri does provide some cursory guidance on what information the AI may be able to provide,  “Some things you can ask me:”

With only voice recognition input, use the Voice Driven Menu Navigation & Selection approach as described below.

Voice Driven, Menu Navigation and Selection

The current AI personal assistant, abstraction layer may be too abstract for some users.  The difference between these two commands:

  • Play The Rolling Stones song Sympathy for the Devil.
    • Has the benefit of natural language, and can handle simple tasks, like “Call Mom”
    • However, there may be many commands that can be performed by a multitude of installed platform applications.

Verse

  • Spotify.Song.Sympathy for the Devil
    • Enables the user to select the specific application they would like a task to be performed by.
  • Spotify Help
    • A voice driven menu will enable users to understand the capabilities of the AI Assistant.    Through the use of a voice interactive menu, users may ‘drill down’ to the action they desire to be performed. e.g. “Press # or say XYZ”
    • Optionally, the voice menu, depending upon the application, may have a customer service feature, and forward the interaction to the proper [calling or chat] queue.

Update – 9/11/16

  • I just installed Microsoft Cortana for iOS, and at a glance, the application has a leg up on the competition
    • The Help menu gives a fair number of examples by category.  Much better guidance that iOS / Siri 
    • The ability to enter\type or speak commands provides the needed flexibility for user input.
      • Some people are uncomfortable ‘talking’ to their Smartphones.  Awkward talking to a machine.
      • The ability to type in commands may alleviate voice command entry errors, speech to text translation.
      • Opportunity to expand the AI Syntax Parser to include ‘programmatic’ type commands allows the user a more granular command set,  e.g. “Intelligent Command Completion”.  As the capabilities of the platform grow, it will be a challenge to interface and maximize AI Personal Assistant capabilities.

AI Assistant Summarizing Email Threads and Complex Documents

“Give me the 50k foot level on that topic.”
“Just give us the cliff notes.”
“Please give me the bird’s eye view.”

AI Email Thread Abstraction and Summarization

A daunting, and highly public email has landed in your lap..top to respond.  The email thread goes between over a dozen people all across the globe.  All of the people on the TO list, and some on the CC list, have expressed their points about … something.  There are junior technical and very senior business staff on the email.  I’ll need to understand the email thread content from the perspective of each person that replied to the thread.  That may involve sifting through each of the emails on the thread.  Even though the people on the emails are English fluent, their response styles may be different based on culture, or seniority of staff (e.g. abstractly written).  Also, the technical folks might want to keep the conversation of the email granular and succinct.
Let’s throw a bit of [AI] automation at this problem.
Another step in our AI personal assistant evolution, email thread aggregation and summarization utilizing cognitive APIs | tools such as what IBM Watson has implemented with their Language APIs.  Based on the documentation provided by their APIs, the above challenges can be resolved for the reader.   A suggestion to an IBM partner for the Watson Cognitive cloud, build an ’email plugin’ if the email product exposes their solution to customization.
A plugin built on top of an email application, flexible enough to allow customization, may be a candidate for Email Thread aggregation and summarization.  Email clients may include IBM Notes, Gmail, (Apple) Mail, Microsoft Outlook, Yahoo! Mail, and OpenText FirstClass.
Add this capability to the job description of AI assistants, such as Cortana, Echo, Siri, and Google Now.   In fact, this plug-in may not need the connectivity and usage of an AI assistant, just the email plug-in interacting with a suite of cognitive cloud API calls.

AI Document Abstraction and Summarization

A plug in may also be created for word processors such as Microsoft Word.   Once activated within a document, a summary page may be created and prefixed to the existing document. There are several use cases, such as a synopsis of the document.
With minimal effort from human input, marking up the content, we would still be able to derive the  contextual metadata, and leverage it to create new sentences, paragraphs of sentences.
Update:
I’ve not seen an AI Outlook integration in the list of MS Outlook Add-ins that would bring this functionality to users.

Building Apps Incorporating the AI Power of IBM Watson’s Cognative Computing Cloud

IBM Watson’s APIs are available today so teams may ramp up quickly and use IBM’s cognitive computing engine.  From IBM Watson’s site, it seems like anyone may build against their cognitive computing platform.  In addition,  your team may submit to be ‘Featured’ in their application Gallery.  Explore the library of featured applications produced by this partnership.  At the time of this writing, there were 14 applications.
Several of these apps have been created by IBM to showcase their technology.  IBM Watson APIs are categorized into ‘Services’ used:
  • Dialog
  • Natural Language Classifier
  • AlchemyData News
  • Personality Insights
  • Tradeoff Analytics
  • Speech to Text
  • Language Translation
  • Text to Speech
  • Visual Recognition
  • Concept Insights
  • Relationship Extraction

They sound like AI,BI comprehensive services, but in full disclosure, I’ve not read though the API docs available by IBM.  It can be found here, grouped by IBM Watson’s Cognitive Services.

One of the applications powered by IBM Watson in their gallery is a “News Explorer”, which leverages the Service ‘AlchemyData News’.
The app runs in a browser, and consists of 5 main User Interface components.  The centrally placed, “News Network” widget similar to a mind map, correlates articles, companies, organizations, and people.  Visually it displays these components and their relationships in groupings similar to a relationship tree.
 News Network
The left side of the screen has a table called ‘Details’, one column with short descriptions of the stories.  From the UI perspective, it enables users to follow the data from left to right, from details to graphical representations.
Details
The right most side of the screen contains a world map leveraged as a heat map in which all the News is derived.
Locations
Right under the ‘Locations’ widget, there is a ‘Topics’ tag cloud.
TopicsTags
I encourage you to check out the News application, click here.
In addition to UI drill down within the widgets, there is a comprehensive search capacity.
Are you ready to compete with Siri, or Cortana, or build your own Expert solution?  Looks like IBM is empowering you to do so!

Cooking Wars: Popularizing the HoloLens before it’s Released.

An aspiring Chief or Cook armed with a HoloLens on a special edition of the Food Network show Chopped?  How well would HoloLens and Human come together to create a brilliant dish?
At the TV show’s core rules, each contestant must come up with a dish to be served to the Judges.  The caveat, the Chiefs must use all the ingredients from a ‘blind’ basket.  To enable a Chief with a HoloLens would instantly give the contestant a potentially ‘unfair’ advantage.  Bringing a computer with Internet access and your list of digital recipes would, on the ‘Surface’, be an equivalent advantage.
If the producers of Chopped want to level the playing field,  why not allow the other contestants use of a Microsoft Surface, continuing along the same lines of providing a HoloLens to one, or potentially all of the Chiefs.
‘Adhoc’ cooking with a HoloLens, the user may:
  • Search libraries of food recipes, filtered by the basket ingredients.   HoloLens uses object recognition to identify each of the items taken out of the basket.  Chiefs should not have to ‘say’ or ‘input’ the ingredients to the HoloLens.  Take every advantage to speed up, not slow down interaction of Chef and machine ‘working together’
  • A step by step walk through to execute the recipe, HoloLens and human working together. e.g. HoloLens highlights the salt on the user’s field of vision.  HoloLens articulates what is needed and when, like a tutor over your shoulder.
  • Recipes may have a ‘pause’ to allowremind the Chief to ‘check the food’, and provide feedback to the HoloLens.  AI on the HoloLens may indicate back to the Chief to action, such as the Chief saying it tastes too XYZ, so HoloLens responds, add NN of Salt.
  • HoloLens may also state reminders such as ‘you should be plating your food by now’,  or you haven’t added this ingredient yet.
  • The HoloLens may guide the ‘food plating’ process, almost like an empty puzzle being populated.

Note: Microsoft is not responsible for any  accidental cuts. 🙂

What’s Next?

IBM’s Watson goes into battle on Bravo’s Top Chef  with the aid of any household  cook, and an Augmented Reality (AR) Headset, such as the HoloLens.