AI Personal Assistant Needs Remedial Guidance for their Users

Providing Intelligent ‘Code’ Completion

At this stage in the application platform growth and maturity of the AI Personal Assistant, there are many commands and options that common users cannot formulate due to a lack of knowledge and experience .

A key usability feature for many integrated development environments (IDE) are their capability to use “Intelligent Code Completion” to guide their programmers to produce correct, functional syntax. This feature also enables the programmer to be unburdened by the need to look up syntax for each command reference, saving significant time.  As the usage of the AI Personal Assistant grows, and their capabilities along with it, the amount of “command and parameters” knowledge required to use the AI Personal Assistant will also increase.

AI Leveraging Intelligent Command Completion

For each command parameter [level\tree], a drop down list may appear giving users a set of options to select for the next parameter. A delimiter such as a period(.) indicates to the AI Parser another set of command options must be presented to the person entering the command. These options are typically in the form of drop down lists concatenated to the right of the formulated commands.

AI Personal Assistant Language Syntax

Adding another AI parser on top of the existing syntax parser may allow commands like these to be executed:

  • Abstraction (e.g. no application specified)
    • Order.Food.Focacceria.List123
    • Order.Food.FavoriteItalianRestaurant.FavoriteLunchSpecial
  • Application Parser
    • Seamless.Order.Food.Focacceria.Large Pizza

These AI command examples uses a hierarchy of commands and parameters to perform the function. One of the above commands leverages one of my contacts, and a ‘List123’ object.  The ‘List123’ parameter may be a ‘note’ on my Smartphone that contains a list of food we would like to order. The command may place the order either through my contact’s email address, fax number, or calling the business main number and using AI Text to Speech functionality.

All personal data, such as Favorite Italian Restaurant,  and Favorite Lunch Special could be placed in the AI Personal Assistant ‘Settings’.  A group of settings may be listed as Key-Value pairs,  that may be considered short hand for conversations involving the AI Assistant.

A majority of users are most likely unsure of many of the options available within the AI Personal assistant command structure. Intelligent command [code] completion empowers users with visibility into the available commands, and parameters.

For those without a programming background, Intelligent “Command” Completion is slightly similar to the autocomplete in Google’s Search text box, predicting possible choices as the user types. In the case of the guidance provided by an AI Personal Assistant the user is guided to their desired command; however, the Google autocomplete requires some level or sense of the end result command. Intelligent code completion typically displays all possible commands in a drop down list next to the constructor period (.). In this case the user may have no knowledge of the next parameter without the drop down choice list.  An addition feature enables the AI Personal Assistant to hover over one of the commands\parameters to show a brief ‘help text’ popup.

Note, Microsoft’s Cortana AI assistant provides a text box in addition to speech input.  Adding another syntax parser could be allowed and enabled through the existing User Interface.  However, Siri seems to only have voice recognition input, and no text input.

Is Siri handling the iOS ‘Global Search’ requests ‘behind the scenes’?  If so, the textual parsing, i.e. the period(.) separator would work. Siri does provide some cursory guidance on what information the AI may be able to provide,  “Some things you can ask me:”

With only voice recognition input, use the Voice Driven Menu Navigation & Selection approach as described below.

Voice Driven, Menu Navigation and Selection

The current AI personal assistant, abstraction layer may be too abstract for some users.  The difference between these two commands:

  • Play The Rolling Stones song Sympathy for the Devil.
    • Has the benefit of natural language, and can handle simple tasks, like “Call Mom”
    • However, there may be many commands that can be performed by a multitude of installed platform applications.

Verse

  • Spotify.Song.Sympathy for the Devil
    • Enables the user to select the specific application they would like a task to be performed by.
  • Spotify Help
    • A voice driven menu will enable users to understand the capabilities of the AI Assistant.    Through the use of a voice interactive menu, users may ‘drill down’ to the action they desire to be performed. e.g. “Press # or say XYZ”
    • Optionally, the voice menu, depending upon the application, may have a customer service feature, and forward the interaction to the proper [calling or chat] queue.

Update – 9/11/16

  • I just installed Microsoft Cortana for iOS, and at a glance, the application has a leg up on the competition
    • The Help menu gives a fair number of examples by category.  Much better guidance that iOS / Siri 
    • The ability to enter\type or speak commands provides the needed flexibility for user input.
      • Some people are uncomfortable ‘talking’ to their Smartphones.  Awkward talking to a machine.
      • The ability to type in commands may alleviate voice command entry errors, speech to text translation.
      • Opportunity to expand the AI Syntax Parser to include ‘programmatic’ type commands allows the user a more granular command set,  e.g. “Intelligent Command Completion”.  As the capabilities of the platform grow, it will be a challenge to interface and maximize AI Personal Assistant capabilities.

2016 Olympics Rating are Down? Don’t Blame Streaming!

The 2016 Olympic opening ceremonies had just started, and I thought briefly about events I wanted to see.  I’m not a huge fan of the Olympics mostly because of the time commitment.  However, if I happen to be in front of the TV when the events are on, depending upon the event, I’m happy to watch, and can get drawn in easily.

As the Olympics unfolded, I caught a few minutes of an event here and there, just by happening to be in front of a TV.  Searching for any particular event never crossed my mind, even with the ease and power behind several powerful search engines like Bing and Google. The widgets built into search engine’s results showing Olympic standings in line with other search results was a great time saver.

However, why oh why didn’t the broadcasting network NBC create a calendar of Olympic 2016  events that can easily be imported into either Google Calendar, or Microsoft Outlook?  Even Star Trek fans are able to add a calendar to their Google Calendar for Star Dates.

Olympic ratings are hurting?  Any one of these organizations could have created a shared calendar for all or a subset of Olympic  events. Maybe you just want a calendar that shows all the aquatic events?

Olympic Team Sponsors from soda to fast food, why oh why did you paint your consumer goods with pictures of Javelin throwers and Swimmers, but didn’t put a QR code on the side of your containers that directs consumers to your sponsored team’s calendar schedule “importable” into Google Calendar, or Microsoft Outlook?

If sponsors, or the broadcasting network, NBC, would have created these shareable calendars, you now would had entered the personal calendars of the consumer.  A calendar entry pop-up may not only display what current competition is being fought, the body of the event may also contain [URL] links to stream the event live.  The body of the event may also contain links to each team player’s stats, and other interesting facts relating to the event.

Also, if a Team Sponsor is the one creating the custom calendar for the Olympic Events, like USA Swimming’s sponsor Marriott , the streaming live video events may now be controlled by the Sponsor, yes, all advertising during the streaming session would be controlled by the the Sponsor.  All Marriott!  The links in the team sponsor calendar entries may not only have their own streaming links to the live events, but include any feature rich, relevant related content.

There is the small matter of broadcast licensing Olympic Broadcasting Services (OBS)  and broadcaster exclusivity, but hey, everything is negotiable.  Not sure traditional broadcasting rules should apply in a world of video streaming.

All the millions sponsors spend, for an IT Project that could cost a fraction of their advertising budget, and add significant ROI, it boggles the mind why every sponsor isn’t out there doing this or something similar right now.  The tech is relatively inexpensive, and readily available, so why not now?  If you know of any implementations, please drop me a note.

One noted exception, the “Google app” [for the iPhone] leverages alerts for all types of things such as a warning on traffic conditions for your ride home to … the start of the Women’s beam Gymnastics Olympic event.   Select the alert, and opens up a ‘micro’ portal with people competing in the event, a detailed list of athlete profiles, including picture, country of origin, and metals won.  There is also a tab showing the event future schedule.

Microsoft Flow – Platform Review

It looks like Microsoft created a generic workflow platform, product independent.

Microsoft has software solutions, like MS Outlook with an [email] rules engine built into Outlook.  SharePoint has a workflow solution within the Sharepoint Platform, typically governing the content flowing through it’s system.

Microsoft Flow is a different animal.  It seems like Microsoft has built a ‘generic’ rules engine for processing almost any event.  The Flow product:

  1. Start using the product from one of two areas:  a) “My Flows” where I may view existing and create new [work]flows. b) “Activity”, that shows “Notifications” and “Failures”
  2. Select “My Flows”, and the user may “Create [a workflow] from Blank”,  or “Browse Templates”.  MSFT existing set of templates were created by Microsoft, and also by a 3rd party implying a marketplace.
  3. Select “Create from Blank” and the user has a single drop down list of events, a culmination events across Internet products. There is an implication there could be any product, and event “made compatible” with MSFT Flows.
    1. The drop down list of events has a format of “Product – Event”.  As the list of products and events grow, we should see at least two separate drop down lists, one for products, and a sub list for the product specific events.
    2. Several Example Events Include:
      1. “Dropbox – When a file is created”
      2. “Facebook – When there is a new post to my timeline”
      3. “Project Online – When a new task is created”
      4. “RSS – When a feed item is published”
      5. “Salesforce – When an object is created”
    3. The list of products as well as there events may need a business analyst to rationalize the use cases.
  4. Once an Event is selected, event specific details may be required, e.g. Twitter account details, or OneDrive “watch” folder
  5. Next, a Condition may be added to this [work]flow,  and may be specific to the Event type, e.g. OneDrive File Type properties [contains] XYZ value.  There is also an “advanced mode” using a conditional scripting language.
  6. There is “IF YES” and “IF NO” logic, which then allows the user to select one [or more] actions to perform
    1. Several Action Examples Include:
      1. “Excel – Insert Rows”
      2. “FTP – Create File”
      3. “Google Drive – List files in folder”
      4. “Mail – Send email”
      5. “Push Notification – Send a push notification”
    2. Again, it seems like an eclectic bunch of Products, Actions, and Events strung together to have a system to POC.
  7. The Templates list, predefined set of workflows that may be of interest to anyone who does not want to start from scratch.   The UI provides several ways to filter, list, and search through templates.

Applicable to everyday life, from an individual home user, small business, to the enterprise.  At this stage the product seems in Beta at best, or more accurately, just after clickable prototype.  I ran into several errors trying to go through basic use cases, i.e. adding rules.

Despite the “Preview” launch, Microsoft has showed us the power in [work]flow processing regardless of the service platform provider, e.g.  Box, DropBox, Facebook, GitHub, Instagram, Salesforce, Twitter, Google, MailChimp, …

Microsoft may be the glue to combine service providers who may / expose their services to MSFT Flow functionality.

Create from Blank - Select Condition
Create from Blank – Select Condition

 

Create Rule from Template
Create Rule from Template
Create from Blank Rule Building UI
Create from Blank Rule Building UI

 

Update June 28th, 2016:

Opportunities for Event, Condition, Action Rules

  • Transcoding [cloud] Services
  • [IBM Watson] Cognitive APIs
    • e.g. Language:Translation; E.g.2. Visual Recognition;
  • WordPress – Create a Post
    • New text file dropped in specific folder on Box, DropBox, etc. being ‘monitored’ by MSFT flow [?] Additional code required by user for ‘polling’ capabilities
    • OR new text file attached, and emailed to specific email account folder ‘watched’ by MSFT Flow.
    • Event triggers – Automatic read of new text file
      • stylizing may occur if HTML coding used
    • Action – Post to a Blog
  • ‘ANY’ Event occurs, a custom message is sent using Skype for a single or group of Skype accounts;
    • On several ‘eligible’ events, such as “File Creation” into Box,  the file (or file shared URL) may be sent to the Skype account.
  • ‘ANY’ Event occurs, a custom mobile text message is sent to a single or group of phone numbers.
  • Event occurs for “File Creation” e.g. into Box; after passing a “Condition”, actions occur:
    • IBM Watson Cognitive API, Text to Speech, occurs, and the product of the action is placed in the same Box folder.
  • Action: Using Microsoft Edge (powered by MSN), in the “My news feed” tab, enable action to publish “Cards”, such as app notifications

Challenges \ Opportunities \ Unknowns

  • 3rd party companies existing, published [cloud; web service] APIs may not even need any modification to integrate with Microsoft Flow; however, business approval may be required to use the API in this manner,
  • It is unclear re: Flow Templates need to be created by the product owner, e.g. Telestream, or knowledgeable third party, following the Android, iOS, and/or MSFT Mobile Apps model.
  • It is unclear if the MSFT Flow app may be licensed individually in the cloud, within the 365 cloud suite, or offered for Home and\or Business?

Cloud Storage: Ingestion, Management, and Sharing

Cloud Storage Solutions need differentiation that matters, a tipping point to select one platform over the other.

Common Platforms Used:

Differentiation may come in the form of:

  • Collaborative Content Creation Software, such as DropBox Paper enables individuals or teams to produce content, all the while leveraging the Storage platform for e.g. version control,
  • Embedded integration in a suite of content creation applications, such as Microsoft Office, and OneDrive.
  • Making the storage solution available to developers, such as with AWS S3, and Box.  Developers may create apps powered by the Box Platform or custom integrations with Box
  • iCloud enables users to backup their smartphone, as well tightly integrating with the capture and sharing of content, e.g. Photos.

Cloud Content Lifecycle Categories:

  • Content Creation
    • 3rd Party (e.g. Camera) or Integrated Platform Products
  • Content Ingestion
    • Capture Content and Associated Metadata
  • Content Collaboration
    • Share, Update and Distribution
  • Content Discovery
    • Surface Content; Searching and Drill Down
  • Retention Rules
    • Auto expire pointer to content, or underlying content

Cloud Content Ingestion Services:

Cloud Ingestion Services
Cloud Ingestion Services

Applying Gmail Labels Across All Google Assets: Docs, Photos, Contacts + Dashboard, Portal View

Google applications contain [types of] assets,  either created within the application, or imported into the application.    In Gmail, you have objects, emails, and Gmail enables users to add metadata to the email in the form of tags or “Labels”.  Labeling emails is a very easy way to organize these assets, emails.   If you’re a bit more organized, you may even devise a logical taxonomy to classify your emails.

An email can also be put into a folder and this is completely different than what we are talking about with labels.  An email may be placed into a folder, and have a parent child folder hierarchy.  Only the name of the folder, and it’s correlations to positions in the hierarchy provide this relational metadata.

For personal use, or for small to medium size businesses, users may want to categorize  all of the Google “objects” from each Google App,  so why Isn’t there the capability to apply labels across all Google App assets?  If you work at a law firm, for example, and have documents in Google Docs, and use Google for email, it would be ideal to leverage a company wide taxonomy, and upon any internal search discover all objects logically grouped in a container by labels.

For each Google object asset, such as email in Gmail, users may apply N number of labels to each Google Object asset.

A [Google] dashboard, or portal view may be used to display and access Google assets across Google applications, grouped by Labels .  A Google Apps “Portal Search” may consist of queries that contain asset labels.  A  relational, Google object repository containing assets across all object types (e.g. Google Docs), may be leveraged to store metadata about each Google asset and their relationships.

A [Google] dashboard, or portal view may be organized around individuals (e.g. personal), teams, or an organization.  So, in a law firm, for example, a case number label could be applied to Google Docs,  Google Photos (i.e. Photos and Videos),  and of course, Gmail.

A relatively simple feature to be implemented with a lot of value for Google’s clients, us?  So, why isn’t it implemented?

One better, when we have facial recognition code implemented in Photos (and Videos), applying Google labels to media assets may allow for correlation of Emails to Photos with a rule based engine.

The Google Search has expanded into the mobile Google app.

Leveraging Google “Cards“, developers may create “Cards” for a single or group of Google assets.   Grouping of Google assets may be applied using “Labels”.   As Google assets go through a business or personal user workflow, additional metadata may be added to the asset, such as additional “Labels”.

Expanding upon this solution,  scripts may be created to “push” assets through a workflow, perhaps using Google Cloud Functions.  Google “Cards” may be leveraged as “the bit” that informs users when they have new items to process in a workflow.

Metadata, or Labels, may be used such as “Document Ready for Legal Review” or “Legal Document Review Completed”.

AI Assistant Summarizing Email Threads and Complex Documents

“Give me the 50k foot level on that topic.”
“Just give us the cliff notes.”
“Please give me the bird’s eye view.”

AI Email Thread Abstraction and Summarization

A daunting, and highly public email has landed in your lap..top to respond.  The email thread goes between over a dozen people all across the globe.  All of the people on the TO list, and some on the CC list, have expressed their points about … something.  There are junior technical and very senior business staff on the email.  I’ll need to understand the email thread content from the perspective of each person that replied to the thread.  That may involve sifting through each of the emails on the thread.  Even though the people on the emails are English fluent, their response styles may be different based on culture, or seniority of staff (e.g. abstractly written).  Also, the technical folks might want to keep the conversation of the email granular and succinct.
Let’s throw a bit of [AI] automation at this problem.
Another step in our AI personal assistant evolution, email thread aggregation and summarization utilizing cognitive APIs | tools such as what IBM Watson has implemented with their Language APIs.  Based on the documentation provided by their APIs, the above challenges can be resolved for the reader.   A suggestion to an IBM partner for the Watson Cognitive cloud, build an ’email plugin’ if the email product exposes their solution to customization.
A plugin built on top of an email application, flexible enough to allow customization, may be a candidate for Email Thread aggregation and summarization.  Email clients may include IBM Notes, Gmail, (Apple) Mail, Microsoft Outlook, Yahoo! Mail, and OpenText FirstClass.
Add this capability to the job description of AI assistants, such as Cortana, Echo, Siri, and Google Now.   In fact, this plug-in may not need the connectivity and usage of an AI assistant, just the email plug-in interacting with a suite of cognitive cloud API calls.

AI Document Abstraction and Summarization

A plug in may also be created for word processors such as Microsoft Word.   Once activated within a document, a summary page may be created and prefixed to the existing document. There are several use cases, such as a synopsis of the document.
With minimal effort from human input, marking up the content, we would still be able to derive the  contextual metadata, and leverage it to create new sentences, paragraphs of sentences.
Update:
I’ve not seen an AI Outlook integration in the list of MS Outlook Add-ins that would bring this functionality to users.

Building AI Is Hard—So Facebook Is Building AI That Builds AI

“…companies like Google and Facebook pay top dollar for some really smart people. Only a few hundred souls on Earth have the talent and the training needed to really push the state-of-the-art [AI] forward, and paying for these top minds is a lot like paying for an NFL quarterback. That’s a bottleneck in the continued progress of artificial intelligence. And it’s not the only one. Even the top researchers can’t build these services without trial and error on an enormous scale. To build a deep neural network that cracks the next big AI problem, researchers must first try countless options that don’t work, running each one across dozens and potentially hundreds of machines.”


This article represents a true picture of where we are today for the average consumer and producer of information, and the companies that repurpose information, e.g. in the form of advertisements.  
The advancement and current progress of Artificial Intelligence, Machine Learning, analogously paints a picture akin to the 1970s with computers that fill rooms, and accept punch cards as input.
Today’s consumers have mobile computing power that is on par to the whole rooms of the 1970s; however, “more compute power” in a tinier package may not be the path to AI sentience.  How AI algorithm models are computed might need to take an alternate approach.  
In a classical computation system, a bit would have to be in one state or the other. However quantum mechanics allows the qubit to be in a superposition of both states at the same time, a property which is fundamental to quantum computing.
The construction, and validation of Artificial Intelligence, Machine Learning, algorithm models should be engineered on a Quantum Computing framework.

Time Lock Encryption: Seal Files in Cloud Storage

Is there value in providing users the ability to apply “Time Lock Encryption” to files in cloud storage?  Files are securely uploaded by their Owner.  After upload no one, including the Owner, may decrypt and access / open the file(s).   Only after the date and time provided for the time lock passes, files will be decrypted, and optionally an action may be taken, e.g. Email a link to the decrypted files to a DL, or a specific person.

Additionally, files might only be decrypted ‘just in time’ and only for the specific recipients who had received the link.  More complex actions may be attached to the time lock release such as script execution using a simple set of rules as defined by the file Owner.

The encryption should be the highest available as defined by the regional law in which the files reside.  Note: issue with cloud storage and applicable regional laws, I.e. In the cloud.

Already exists as a 3rd party plugin to an existing cloud solution?Please send me a link to the cloud integration product / plug in.

AI Personal Assistants are “Life Partners”

Artificial Intelligent (AI)  “Assistants”, or “Bots” are taken to the ‘next level’ when the assistant becomes a proactive entity based on the input from human intelligent experts that grows with machine learning.

Even the implication of an ‘Assistant’ v.  ‘Life Partner’ implies a greater degree of dynamic, and proactive interaction.   The cross over to becoming ‘Life Partner’ is when we go ‘above and beyond’ to help our partners succeed, or even survive the day to day.

Once we experience our current [digital, mobile] ‘assistants’ positively influencing our lives in a more intelligent, proactive manner, an emotional bond ‘grows’, and the investment in this technology will also expand.

Practical Applications Range:

  • Alcoholics Anonymous Coach , Mentor – enabling the human partner to overcome temporary weakness. Knowledge,  and “triggers” need to be incorporated into the AI ‘Partner’;  “Location / Proximity” reminder if person enters a shopping area that has a liquor store.  [AI] “Partner” help “talk down”
  • Understanding ‘data points’ from multiple sources, such as alarms,  and calendar events,  to derive ‘knowledge’, and create an actionable trigger.
    • e.g. “Did you remember to take your medicine?” unprompted; “There is a new article in N periodical, that pertains to your medicine.  Would you like to read it?”
    • e.g. 2 unprompted, “Weather calls for N inches of Snow.  Did you remember to service your Snow Blower this season?”
  • FinTech – while in department store XYZ looking to purchase Y over a certain amount, unprompted “Your credit score indicates you are ‘most likely’ eligible to ‘sign up’ for a store credit card, and get N percentage off your first purchase”  Multiple input sources used to achieve a potential sales opportunity.

IBM has a cognitive cloud of AI solutions leveraging IBM’s Watson.  Most/All of the 18 web applications they have hosted (with source) are driven by human interactive triggers, as with the “Natural Language Classifier”, which helps build a question-and-answer repository.

There are four bits that need to occur to accelerate adoption of the ‘AI Life Partner’:

  1. Knowledge Experts, or Subject Matter Experts (SME) need to be able to “pass on” their knowledge to build repositories.   IBM Watson Natural Language Classifier may be used.
  2. The integration of this knowledge into an AI medium, such as a ‘Digital Assistant’ needs to occur with corresponding ‘triggers’ 
  3. Our current AI ‘Assistants’ need to become [more] proactive as they integrate into our ‘digital’ lives, such as going beyond the setting of an alarm clock, hands free calling, or checking the sports score.   Our [AI] “Life Partner” needs to ‘act’ like buddy and fan of ‘our’ sports team.  Without prompting, proactively serve up knowledge [based on correlated, multiple sources], and/or take [acceptable] actions.
    1. E.g. FinTech – “Our schedule is open tonight, and there are great seats available, Section N, Seat A for ABC dollars on Stubhub.  Shall I make the purchase?”
      1. Partner with vendors to drive FinTech business rules.
  4. Take ‘advantage’ of more knowledge sources, such as the applications we use that collect our data.  Use multiple knowledge sources in concert, enabling the AI to correlate data and propose ‘complex’ rules of interaction.

Our AI ‘Life Partners’ may grow in knowledge, and mature the relationship between man and machine.   Incorporating derived rules leveraging machine learning, without input of a human expert, will come with risk and reward.

Smart Solutions

Powered by "To Do List Member"
Skip to toolbar