Tag Archives: Twitter

Platform Independent AI Model for Images: AI Builder, Easily Utilized by 3rd Party Apps

With all the discourse on OpenAI’s ChatGPT and Natural language processing (NLP), I’d like to steer the conversation toward images/video and object recognition. This is another area in artificial intelligence primed for growth with many use cases. Arguably, it’s not as shocking, bending our society at its core, creating college papers with limited input, but Object Recognition can seem “magical.” AI object recognition may turn art into science, as easy as AI reading your palm to tell your future. AI object recognition will bring consumers more data points from which Augmented Reality (AR) overlays digital images within an analog world of tangible objects.

Microsoft’s AI Builder – Platform Independent

Microsoft’s Power Automate AI [model] Builder has the functionality to get us started on the journey of utilizing images, tagging them with objects we recognize, and then training the AI model to recognize objects in our “production” images. Microsoft provides tools to build AI [image] models (library of images with human, tagged objects) quickly and easily. How you leverage these AI models is the foundation of “future” applications. Some applications are already here, but not mass production. The necessary ingredient: taking away the proprietary building of AI models, such as in social media applications.

In many social media applications, users can tag faces in their images for various reasons, mostly who to share their content/images with. In most cases, images can also be tagged with a specific location. Each AI image/object model is proprietary and not shared between social media applications. If there was a standards body, an AI model could be created/maintained outside of the social media applications. Portable AI object recognition models with a wide array of applications that support it’s use, such as social media applications. Later on, we’ll discuss Microsoft’s AI Model builder, externalized from any one application, and because it’s Microsoft, it’s intuitive. ūüôā

An industry standards body could collaborate and define what AI models look like their features, and most importantly, the portability formats. Then the industry, such as social media apps, can elect to adopt features that are and are not supported by their applications.

Use Cases for Detecting Objects in Images

Why doesn’t everyone have an AI model containing tagged objects within images and videos of the user’s design? Why indeed.

1 – Brands / Product Placement from Content Creators

Just about everyone today is a content creator, producing images and videos for their own personal and business social media feeds, Twitter, Instagram, Snap, Meta, YouTube, and TikTok, to name a few. AI models should be portable enough to integrate with social media applications where tags could be used to identify branded apparel, jewelry, appliances, etc. Tags could also contain metadata, allowing content consumers to follow tagged objects to a specified URL. Clicks and the promotion of products and services.

2 – Object Recognition for Face Detection

Has it all been done? Facebook/Meta, OneDrive, iCloud, and other services have already tried or are implementing some form of object detection in the photos you post. Each of these existing services implements object detection at some level:

  • Identify the faces in your photos, but need you to tag those faces and some “metadata” will be associated with these photos
  • Dynamically grouping/tagging all “Portrait” pictures of a specific individual or events from a specific day and location, like a family vacation.
  • Some image types, JPEGs, PNG, GIF, etc., allow you to add metadata to the files on your own, e.g. so you can search for pictures on the OS level of implementation.
3 – Operational Assistance through object recognition using AR
  • Constructing “complex” components in an assembly line where Augmented Reality (AR) can overlay the next step in assembly with the existing object to help transition the object to the next step in assembly.
  • Assistance putting together IKEA furniture, like the assembly line use case, but for home use.
  • Gaming, everything from Mario Kart Live to Light Saber duels against the infamous Darth Vader.
4 – Palm Reading and other Visual Analytics
  • Predictive weather patterns
5 – Visual Search through Search Engines and Proprietary Applications with Specific Knowledge Base Alignment
  • CoinSnap iPhone App scans both sides of the coin and then goes on to identify the coin, building a user’s collection.
  • Microsoft Bing’s Visual Search and Integration with MSFT Edge
  • Medical Applications, Leveraging AI, e.g., Image Models – Radiology
Radiology – Reading the Tea Leaves

Radiology builds a model of possible issues throughout the body. Creating images with specific types of fractures can empower the autodetection of any issues with the use of AI. If it was a non-proprietary model, radiologists worldwide could contribute to that AI model. The displacement of radiology jobs may inhibit the open non-proprietary nature of the use case, and the AI model may need to be built independently of open input from all radiologists.

Microsoft’s AI Builder – Detect Objects in Images

Microsoft’s AI model builder can help the user build models in minutes. Object Detection, Custom Model, Detect custom objects in images is the “template” you want to use to build a model to detect objects, e.g. people, cars, anything, rather quickly, and can enable users to add images (i.e. train model) to become a better model over time.

Many other AI Model types exist, such as Text Recognition within images. I suggest exploring the Azure AI Models list to fit your needs.

Current, Available Data Sources for Image Input

  • Current Device
  • SharePoint
  • Azure BLOB

Wish List for Data Sources w/Trigger Notifications

When a new image is uploaded into one of these data sources, a “trigger” can be activated to process the image with the AI Model and apply tags to the images.

  • ADT – video cam
  • DropBox
  • Google Drive
  • Instagram
  • Kodak (yeah, still around)
  • Meta/Facebook
  • OneDrive
  • Ring -video cam
  • Shutterfly
  • Twitter

Get Started: Power Automate, Premium Account

Login to Power Automate with your premium account, and select “AI Builder” menu, then the “Models” menu item. The top left part of the screen, select “New AI Model,” From the list of model types, select “Custom Model, Object Detection”Detect Custom Objects in Images.”

AI Builder - Custom Model
AI Builder – Custom Model

It’s a “Premium” feature of Power Automate, so you must have the Premium license. Select “Get Started”,. The first step is to “Select your model’s domain”, there are three choices, so I selected “Common Objects” to give me the broadest opportunity. Then select “Next”.

AI Builder - Custom Model - Domain
AI Builder – Custom Model – Domain

Next, you need to select all of the objects you want to identify in your images. For demonstration purposes, I added my family’s first names as my objects to train my model to identify in images.

AI Builder - Custom Model - Objects for Model
AI Builder – Custom Model – Objects for Model

Next, you need to “Add example images for your objects.” Microsoft’s guidance is “You need to add at least 15 images for each object you want to detect.” Current data sources include:

Add Images
AI Model – Add Images

I added the minimum recommended images, 15 per object, two objects, 30 images of my family, and random pics over the last year.

Once uploaded, you need to go through each image, draw a box around the image’s objects you want to tag, and then select the object tag.

Part 2 – Completing the Model and its App usage.

Twitter’s New Tip Jar for Charity Funraising

I’m excited to enable the new Tip Jar feature for fundraising for charities. Twitter users should be able to create a campaign to raise money for a charity boasted with Tweets related to the charity.

The Twitter Tip Jar should allow users to raise money, goal oriented with a tweatable bar chart that indicates progress, current money raised, and target goals.

Twitter Tip Chart should allow multiple fundraising initiatives in parallel, and tweet tags indicate which charity donation.

Twitter Advertising should allow tweet goal oriented tip jar fundraising, and extend the Twitter Advertising model to pause funding tweets based on daily fundraising goals.

Level Up Social Media with Microsoft Power Automate

Create Automated Workflows with Microsoft Power Automate

I’ve been using this powerful workflow automation platform since it was called Microsoft Flow, and was free for low volume usage. Essentially, users can pick any sources of data, create triggers, transform data to a multitude of target systems, and notify through a multitude of opportunities, such as eMail and push notifications. The platform is boundless through “Connectors” to just about any 3rd party platforms from SalesForce to an Oracle database. The basic plan after the free trial is 15 USD per month.

Connectors and Templates: Ready, Set, Go

1st, define your connectors, such as your Google Email account connection details, and your Twitter account information. 2nd, select from one of the many “out of the box” predefined templates, such as from the “Social Media” category.

MSFT Power Automate Templates
MSFT Power Automate Templates

Twitter Use Cases – Configure In Minutes

Once you’ve signed up for the Power Automate SaaS platform, you can start creating workflows in minutes. At first I used the “Templates”, but it is much easier to create workflows from scratch. Here are a few opportunities foe getting started

Retweet based on Tweet Search Criteria

  • Define what tweets you would like to retweet using query search criteria of words, a combination of hashtags, phrases with simple AND and OR logic.
  • Optionally, add a condition before performing an action within the workflow. In this case, we can allow the retweet only if the retweet count is greater than N retweets.
  • Select the returned Tweet ID to perform the “Retweet” action
  • Optionally, add notifications, such as emailing yourself each time you retweet and include elements of the tweet within your Email, such as the tweet text, tweet user ID, or a dozen of other tweet elements.

Catalog Tweets when they meet your Tweet Criteria

  • Define what tweets you would like to store in your “data” repository.
  • Select from one of a multitude of data targets ranging from Excel spreadsheets, Google Sheets, SQL Server, Oracle Database, and dozens of other repositories.
  • Based on the data target, perform the mapping of available tweet elements to the data target fields, such as a database, table, and fields.

And Beyond

Microsoft Power Automate can do automated workflows beyond the social media capabilities highlighted here. I have a wish list of “Triggers” and “Actions” not yet supported by the platform. I’d like to have the same “Trigger” criteria we have with Twitter extended to LinkedIn, and trigger LinkedIn posts based on query criteria, extract, and load into en external data source.

Twitter Trolls caused Salesforce to Walk Away from Deal? Google reCAPTCHA to the Rescue!?

According to CNBC’s “Mad Money” host Jim Cramer, Salesforce¬†was turned off by a more fundamental problem that’s been hurting Twitter for years: trolls.

“What’s happened is, a lot of the bidders are looking at people with lots of followers and seeing the hatred,” Cramer said on CNBC’s “Squawk on the Street,” citing a recent conversation with Benioff. “I know that the haters reduce the value of the company…I know that Salesforce was very concerned about this notion.”

…Twitter’s troll problem isn’t anything new if you’ve been following the company for a while.”

Source: Twitter trolls caused Salesforce to walk away from deal – Business Insider

Anyone with a few neurons will recognize that bots on Twitter are a huge turnoff in some cases. ¬†I like periodic famous quotes as much as the next person, but it seems like bots have invaded Twitter for a long time, and becomes a detractor¬†to using the platform. ¬†The solution in fact is quite easy,¬†reCAPTCHA. ¬†a web application that determines if the user is a human and not a robot. ¬†Twitter users should be required to use¬†an integrated reCAPTCHA Twitter DM, and/or as a “pinned”reCAPTCHA tweet that sticks to the¬†top of your feed,¬†¬†once a calendar week, and go through the “I’m not a robot” quick and easy process.

Additionally, an AI rules engine may identify particular patterns of Bot activity, flag it, and force the user to go through the Human validation process within 24 hours. ¬†If users try to ‘get around’ the Bot\Human identification process, ¬†maybe by tweaking their tweets, Google may employ AI machine learning algorithms to feed the “Bot” AI rules engine patterns.

Every Twitter user identified as “Human” would¬†have the picture of the “Vitruvian Man” by¬† Leonardo da Vinci miniaturized, and placed next to the “Verified Account” check mark. ¬†Maybe there’s a fig leaf too.

In addition, the user MAY¬†declare it IS a bot, and there are certainly valid reasons to utilize bots. ¬†Instead of the “Man” icon, Twitter may allow users to pick the bot icon, including the character¬†from the TV show “Futurama”, Bender miniaturized. ¬†Twitter could¬†collect additional information on Bots for enhanced user experience, e.g. categories and subcategories

reCAPTCHA is owned by Google, so maybe, in some far out distant universe, a Doppelgänger Google would buy Twitter, and either phase out or integrate G+ with Twitter.

If trolls/bots¬†are¬†such a huge issue, why hasn’t Twitter addressed¬†it? ¬†What is Google using to deal with the issue?

The prescribed method seems too easy and cheap to implement, so I must be missing something.  Politics maybe?  Twitter calling upon a rival, Google (G+) to help craft a solution?

Microsoft Flow – Platform Review

It looks like Microsoft created a generic workflow platform, product independent.

Microsoft has software solutions, like MS Outlook with¬†an¬†[email] rules engine built into Outlook. ¬†SharePoint has a workflow solution within the Sharepoint Platform, typically governing the content flowing through it’s system.

Microsoft Flow is a different animal. ¬†It seems like¬†Microsoft has built a ‘generic’ rules engine for processing almost any event. ¬†The Flow product:

  1. Start using the product from one of two areas: ¬†a) “My Flows” where I may view existing and create new [work]flows. b) “Activity”, that shows “Notifications” and “Failures”
  2. Select “My Flows”, and the user may¬†“Create [a workflow] from Blank”, ¬†or “Browse Templates”. ¬†MSFT¬†existing set of templates were¬†created by Microsoft, and also by a 3rd party implying a marketplace.
  3. Select “Create from Blank” and the user has a single drop down list of events, a culmination¬†events across Internet products. There is an¬†implication¬†there could be any product, and event “made compatible” with MSFT Flows.
    1. The drop down list of events has a format of “Product – Event”. ¬†As the list of products and events grow, we should see at least two separate drop down lists, one for products, and a sub list for the product specific events.
    2. Several Example Events Include:
      1. “Dropbox – When a file is created”
      2. “Facebook – When there is a new post to my timeline”
      3. “Project Online – When a new task is created”
      4. “RSS – When a feed item is published”
      5. “Salesforce – When an object is created”
    3. The list of products as well as there events may need a business analyst to rationalize the use cases.
  4. Once an Event is selected, event specific details may be required, e.g. Twitter account details, or OneDrive “watch” folder
  5. Next, a Condition may be added to this [work]flow, ¬†and may be specific to the Event type, e.g. OneDrive File Type properties [contains] XYZ value. ¬†There is also an “advanced mode” using a conditional scripting language.
  6. There is “IF YES” and “IF NO” logic, which then allows the user to select one [or more] actions to perform
    1. Several Action Examples Include:
      1. “Excel – Insert Rows”
      2. “FTP – Create File”
      3. “Google Drive – List files in folder”
      4. “Mail – Send email”
      5. “Push Notification – Send a push notification”
    2. Again, it seems like an eclectic bunch of Products, Actions, and Events strung together to have a system to POC.
  7. The Templates list, predefined set of workflows that may be of interest to anyone who does not want to start from scratch.   The UI provides several ways to filter, list, and search through templates.

Applicable to everyday life, from an individual home user, small business, to the enterprise.  At this stage the product seems in Beta at best, or more accurately, just after clickable prototype.  I ran into several errors trying to go through basic use cases, i.e. adding rules.

Despite the “Preview” launch, Microsoft has showed us the power in [work]flow processing¬†regardless of the service platform provider, e.g. ¬†Box, DropBox, Facebook, GitHub, Instagram, Salesforce, Twitter, Google, MailChimp, …

Microsoft may be the glue to combine service providers who may / expose their services to MSFT Flow functionality.

Create from Blank - Select Condition
Create from Blank – Select Condition

 

Create Rule from Template
Create Rule from Template

Create from Blank Rule Building UI
Create from Blank Rule Building UI

 

Update June 28th, 2016:

Opportunities for Event, Condition, Action Rules

  • Transcoding [cloud] Services
  • [IBM Watson] Cognitive APIs
    • e.g.¬†Language:Translation; E.g.2. Visual Recognition;
  • WordPress – Create a Post
    • New text file dropped in specific folder on¬†Box, DropBox, etc. being ‘monitored’ by MSFT flow [?] Additional¬†code required by user for ‘polling’ capabilities
    • OR new text file attached, and emailed to specific email account folder ‘watched’ by MSFT Flow.
    • Event triggers – Automatic read of new text file
      • stylizing may occur if HTML coding used
    • Action – Post to a Blog
  • ‘ANY’ Event occurs, a custom message is sent using¬†Skype for a single or group of Skype accounts;
    • On several ‘eligible’¬†events, such as “File Creation” into Box, ¬†the file (or file shared URL) may be sent to the Skype account.
  • ‘ANY’ Event occurs,¬†a custom mobile text message is sent to a single or group of¬†phone numbers.
  • Event occurs for “File Creation” e.g. into Box; after passing a¬†“Condition”, actions occur:
    • IBM Watson Cognitive API,¬†Text to Speech, occurs, and the product of the action is placed in the same Box folder.
  • Action: Using Microsoft Edge (powered by MSN), in the “My news feed” tab, enable action to publish “Cards”, such as app notifications

Challenges \ Opportunities \ Unknowns

  • 3rd party¬†companies existing, published [cloud; web service] APIs may not even need any modification to integrate with Microsoft Flow; however, business approval may be required to use the API in this manner,
  • It is unclear re: Flow Templates need to be created by the product owner, e.g. Telestream, or knowledgeable third party, following the Android, iOS, and/or MSFT Mobile Apps model.
  • It is unclear if the MSFT Flow app may be licensed individually in the cloud, within the 365 cloud suite, or offered for Home and\or Business?

Yahoo’s Opportunity for Success: Entertainment & News Streams

Taking on Tumblir, showing top tweets, trends currently active, is Yahoo going to provide Business Intelligence (BI) for entertainment and news?  I see a paradigm emerging where there are one of several BI widgets to express a visual depliction of real time news and entertainment streams from several sources including active searches, and the popular tags of the moment.

It’s possible Yahoo becomes a dashboard of BI news and entertainment widgets allowing drilling into a deeper level.¬† One widget might be a pie chart, showing a dynamic view of categories of tags for feeds, e.g. world news.¬† Then the user can click and drill into that category showing by regional news.¬† Cloud tags could be another graphical representation deplicting entertainment categories by the celebrity name or topic.

Other BI widgets such as a stop light or rpm speedometer may show hot topics, such as immediate news where the threshold of tweets, searches, posts, exceeds a certain level, the odometer goes from green to red on alert news, tornado warning in your area, or hot topics of any kind. Yahoo needs to evolve their old jump page, and instead of a hogpog of old and new paradigms, proceed with a news and entertainment business intelligence dashboard, default and configurable.

In addition, BI widget streams may be customized based upon a user’s subscription feeds.

Client Relationship Management Systems Integrating Twitter

A few months ago, I had an issue with my Internet Service Provider, griped about it on Twitter and before you know it, the organization has a Twitter search,  and was walking me through a problem.   It was amazing to have that type of personal care and quick response. So thinking about it, why not just add the additional feature to Customer Relationship Management (CRM).

As an example, the CRM system should troll through tweets looking for hashtags which may include company name, any additional issue related content, and then generate a unique ticket ID, which may be a separate queue from the regular CRM.¬† The tweet gets placed as a line item in the comments section appended with date, time and status.¬† A person then reviews the queue to try to identity the actual customer,¬† like following the user and tweeting back, who are you and how can I help suggesting to DM them your customer ID.¬† The companies’ account would be verified, so no trust issues.

If it is a valid customer,  they qualify the interaction, e.g. new service,  tech support,  move the Twitter ticket out of the queue and route a new CRM ticket to the appropriate department, where, they may continue the conversation by phone, or a company authorized chat program.

Sounds cool.