CS Lecture with Rohit Prasad: Vice President and Head Scientist Amazon Alexa
The following day, the group attended the SCS Distinguished Lecture featuring Rohit Prasad, an executive and developer on the Alexa team with Amazon. We were heartened to hear that the focus areas from Amazon were similar to our efforts during the semester, namely their effort to bring conversational context (this open-ended interaction) to Alexa interactions overall. Our main takeaway from this lecture was the fact that the technology isn’t there yet, but isn’t that far away. Perhaps in 5 years, conversational chatbots might be more in reach. In the present, Amazon released Amazon Lex, a way for developers to start experimenting with these very challenges.

Week 16:
This week, we spent preparing our final presentation, as well as our deliverables for the semester’s end. During this time, Amazon announced Echo Show, a new hardware for the Echo system that incorporates a visual screen as well as added utility functions: messaging. This was another element that validated our efforts: messaging is a meaningful and gratifying addition to this platform. Roy’s asynchronous messaging system were in line with Amazon’s direction, as our open ended experience was.  Indeed, the night before our presentation, Amazon deployed a software update to the Alexa app that allowed for voice calls and messages between Echo devices. While we worked to create an entirely audio UX for this space, Amazon relied on its own app, and interfaced with users’ mobile phones.

Final Presentation:
For the final presentation, we chose to focus on our efforts to explore the interactions within the space, and discuss it within the context of Amazon’s current traction and future development. Some of the questions from the audience related to the future of voice recognition in this space, and our team agrees – once voice recognition becomes conversational enough (and aware of conversational context), it will play a large role in UX design, from audio only projects, to VR and more.

Final Play Through:
In the final play throughs, we talked a bit more in depth about our findings overall, and ran through some of the demos we hadn’t shown off to the faculty. They urged us not to let our work disappear, and to this end, we’ve been adding even more information to our handbook, are creating videos showing play throughs of all our demos and plan to create a short article for publication online.

Final Deliverables:
1. Developer’s handbook
2. Alexa Skill Package
– Intruder Alert (Query)
– HaKoo (Syllable Recognition)
– Puns with Friends (Asynchronous Messaging)
– Fortune Teller (Template)

1) Story Demo (Open Ended Questions)
After voice recordings in week 12, Seth spent the majority of his time cutting the clips and creating a track of the entire experience. His goal was to find the story flow, and integrate appropriate SFX and music to accompany the experience. The main challenge was to use music and SFX to elevate the emotions of the story, particularly difficult in a 5 minute experience. In addition, Seth needed to consider the breaks in sound (every time a user interacted) when he composed the music to the experience. He settled for an ebb-and-flow style that would make the breaks for interaction less jarring for the user.

For the story, Sarabeth worked on expanding the number of possible answers in the “open ended” section with our voice actor, and tested throughout with the query system in the editor tool. Now the query system, contrasted with the intent system, allows us to parse sentences and assign it a probability to a given answer. Training it consists of entering in keywords, sentences and other inputs to help it make more accurate assessments. In addition to expanding the depth of the script (for the open ended section), Phan and Roy added the features of transcript retrieval and keyword function. We wanted to add this feature for several reasons. First, we found through playtesting that the first interaction with the user sets the stage for the rest of the experience. For example, having a yes-no question work with “sure” or “definitely” or “Ok” instead of only “Yes” frees up the possibility of nuanced responses. We wanted to set the tone that Alexa was really listening to the user, and decided to try out having her repeat the user’s answer back to them, and classifying it in a category from there. The second choice was including the user’s name. This allows us to immerse the user a little more later on in the story, letting Alexa address him/her by name.

2) Ha Koo (Patterns) Demo
Phan and Seth worked closely to develop the sound and story to support the pattern structure. They decided to proceed with a narrative demo for this interaction for a couple reasons. First, it provided a contrast with the main story demo in giving our guests a light-hearted option. Second, it demonstrated in very basic terms the recognition of syllables (contrasted with the word-based query recognition in the interactions above). The structure of the demo focused on gradually ramping up understanding and teaching of the “ha koo” langauge for users, and ends with the users navigating a conversation entirely with phrases “ha koo” and “koo koo”.

3) Messaging Demo
Roy’s asynchronous messaging demo integrated into a puzzle game, complete with SFX. Originally, we had planned to show this during the Softs walkaround, but were advised to leave room at the end for questions with the faculty.

Our Softs Feedback:
-show all demos to help show range of possible interactions with platform
-explain the limitations of the platform
-make a deliverable that you can share with the dev community
-enter work in the amazon echo competition

For finals, we plan to deliver the package of demos (4 in total) showing off different interaction types. (Template, Navigation, Query/Open-Ended, Pattern, Asynchronous Messaging). In addition to this, we will be creating an Echo developer handbook. Leading up to the ETC Showcase, we will be making revisions to the demos, adding user reprompting (so ensuring that there is enough time for users to respond) and packaging all the demos together via a navigation interaction.

Looking ahead we will be completing our documentation, filming our promotional videos and compiling our records for the end of the semester.

• Need more range in the “open ended” questions when guests speak with our voice acted character
• Shorter answers to the open ended sections, but more pointed to guest questions (for us this means crafting more options for what we anticipate guests will ask)
• First interaction sets the tone for the experience – if Alexa responds appropriately to “Sure” or “I guess” the same as “Yes” we’ve opened the door beyond “yes” or “no”

In addition, the team worked on the short demos for multiplayer messaging, pattern recognition as well as a template answer demo. As a reminder, our goal for the semester to deliver 3-4 small demos, each demonstrating different interactions. Intro – Navigation to demo, 1 – Story, Open-Ended, 2 – Pattern Recognition, language teaching, 3- Asynchronous Messaging Game, Closing – Message to developer. The interactions we do not include will be part of our documentation at the end, showing our design and technical efforts for the semester.

On Friday, Seth brought in 3 drama students from main campus to do voice recording for our story experience (open ended) and the Pattern Recognition (hereafter referred to as “Ha Koo” – the alien language it teaches users). We found their talent really elevated the sound experience of the story. Over the weekend, Seth worked on cutting the clips, and we will plug them into the editor to continue development. Seth composed background music, and we will be looking to secure musician talent next week (perhaps Roy will double as our violinist). Seth’s challenge in composing is twofold: first, something that will offer dramatic richness to the soundscape. Second, something that will work in subtle phrasing, that allows for a logical flow between periods with no sound (user input sections – ie. when the user is asking a question) and sound files.

In Week 13, we will be working on our 4th story demo (voice recording) as well as finalizing our demo package to show at softs.

For our project deliverable, we’ve decided as a group to focus the story on the “open ended” interaction, and provide short demos to showcase the other interactions we’ve developed. What that means for our project delivery is the following:

Project Deliverable:
Campfire Skill:
Navigation to demos (Showcasing our Navigation interaction)

• Main experience demo (Story+ Open-Ended interactions) – More polish than the other demos (voice acting, SFX, music)
• Short demo showcasing Template answer interaction
• Short demo showcasing Pattern-based Voice Recognition
• Short demo showcasing Asynchronous Messaging between Echo devices

In addition to our demo package, we will be documenting our interaction development, design decisions and discoveries in a report that will hope to help future developers in creating new and natural ways for interaction with Smart Assistant devices, as well as design discoveries for audio-only experiences.

Below you’ll find a list of the events with our insights, as well as some of our planned work moving forward.

Week 10
Small Emotion Playtest (Thursday at HCI Playtest Lab) – 4 testers

• Test used 1 dialog line, read with different inflections / emotions
• Discovered that strong emotional responses prompt more user engagement (ie
  , the more rude or outrageous the recording was, the more likely the user would respond)
• Nuanced emotional readings difficult to gauge for listeners

ETC Playtest Day (Saturday at ETC) – 17 testers

• Story length should be 3-5 minutes, not 7-10 minutes (previous)
• Any period longer than 15 seconds of no interaction loses user engagement
• The story topic pertinent
• “Alexa” wake word interrupts skillset, skips through experience
• Open-Ended interaction moment worked for most users
• We also tested the pattern-based voice interaction – users responded positively to a shift in content
• Potential to “learn” a new language (in pattern format) felt fun, and exploration felt fun
• Users didn’t need a “ramping up” of interactions – best to jump right in
• Past experience with the Echo didn’t really affect the results we had

Story Demo

• Recorded with scratch recording from classmates
• Initial structure tested at Playtest Day had combination of passive listening and interaction
• Launched to Echo via editor tool and server
• Further exploration with sound effects, integrating different voice acting as well as other “robotic” voices to sit with Alexa in the experience

Editor Tool Development

• Editor tool revised to include favorites functionality, sound file, intent training and story branching / fail case mapping.
• Able to export directly from tool for live testing
• Some content bugs led to crashing, later corrected in subsequent updates
• Through testing, discovered that 90 second “limitation” has a work around (if you separate distinct sound files form the editor node), but this ended up being a moot point because user attention waned significantly after 15 seconds of passive listening

Asynchronous Messaging Development

• Basic functionality of messaging implemented
• Result: user dictates message, and recipient details. Retrieved message read by Alexa.

Week 11
Story Iteration

• Based on feedback from the playtest day, we shortened the structure to create a 3-5 minute experience, with interactions from the start
• Based on conversations with Anthony Daniels, renewed focus on interactions balanced with performance
• Story read through with Ralph and Scott provided good feedback on tightening up the dialog, structure as well as how to frame the interactions, both open ended and template based

Technical Developments

• Further editor tool development, with added functionality in “open-ended” training (query intent), version control download and upload files, bug fixes and UI adjustments and “bubble set” arrays that allow for varied outputs (so for failcases, we can cycle through different reactions).
• Updated asynchronous messaging system to have the ability to store multiple messages from a single user, as well as add sound emojis to the end of your messages.

Voice Acting

• Seth reached out to contacts on main campus drama dept, received a connection to voice acting talent from the CMU acting class
• Reaching out to voice actors next week, have 9 potential candidates

Alexa Developers Open Hours

• live Q&A for technical discussions and development
• Roy participated, and his notes are here
• One notable discovery: we’ve been having some challenges with Echo shutting off mid-skill. Since we’re able to track that the server wasn’t the source of the issue, the Alexa dev team suggested it might be caused by inconsistent internet connection.

Looking ahead, we’re recording a revised scratch demo for further story testing, and plan to coordinate with voice acting by the end of the week.

This week, Campfire continued to improve our intent system, editor tool and develop further on the asynchronous messaging skill and the nonverbal pattern game skill (think of it as an alien communication game). On the design side, we worked on recording scratch voice acting (recordings will run into Monday) to try and fold in our story draft into Phan’s editor tool. After meeting with Brenda this week, we also recorded assets to try and develop a targeted playtest for how users respond to Alexa when she says the same sentence, but in different emotional tones (think flirtatiously, menacingly, etc).

Short list from the week:

Half Presentation (Video Below)
Twine Story Draft
Scratch Recording – into the weekend
Editor tool updates, intent system updates
Demo using the editor tool

Looking ahead:

Story entered into online editor tool
Scratch recording of story, to test on Echo
Alexa tonal playtest (to test user responses to Alexa’s voice in different emotional tones)
Interaction refinement for in progress interactions (e.g. asynchronous messaging)
Story updates
Training open ended responses
Scheduling for final voice recording
Playtest to Refine Workshop

Before departing for GDC, the team worked on settling up a story and game plan for the time leading up to our Halves presentation. Story-wise, we’ve settled on creating an interaction with War of the Worlds (Radio broadcast) as an inspiration. Sarabeth met with Chris Klug before departing, to talk through some story structure questions, and after a draft of the story is complete, planned a table reading with both Ralph and Scott the following week.

The week after GDC, Seth and Sarabeth fell prey to a conference illness, which derailed some development for the week.

Phan and Roy continued technical development, updating and improving the intent system, as well as creating a story editor for Seth and Sarabeth to use as we start to integrate the design elements into our technical foundation.

This week during Spring Break, the team will be meeting to go over the first draft of the story, create our presentation for next week, and meet with Scott to give a project update.

Log Line:
When a user’s normal interaction with the Echo is interrupted by a stranger hoping to expose corrupt data mining, the user must decide whether to assist the stranger, or alert the system to the intrusion.

Design (adapted from the structure of the “War of the Worlds” radio broadcast):
Act 1: Through the context of (largely) existing Echo interaction types, introduce the character of the stranger as (s)he attempts to “break into” the user’s system and make contact.

Interaction cycles – Using the style and setting of each media type to provide familiarity and foster engagement:

• Podcast interview
• Radio talk show
• News headlines
• Weather report
• Sports summary
• Emergency (Amber) alert
• Account notifications
• Personal messages
• …

Act 2: In a more personal, diary-like writing style, take the user through the results of their choice whether or not to help the stranger, as the story unfolds.

While Sarabeth and Seth began writing versions for the Act 1 components, Phan continued brainstorming and experimenting with other types of Echo interactions, and Roy worked on further developing and refining the Echo intent system.

As a group we discussed how to work best within a narrative-driven production cycle, while still ideating and creating new interaction devices.  How can we continue to explore while affording the writing the needed space, time and consistency for its iterative development?

In this next week, we will explore how additional development can either be used within the current narrative storyline or fit under a larger overall, vignette-style, structure.  A proposal we’ve been considering is to organize these explorations by the styles of engagement users seek with the Echo: to listen, to play and to learn.  Our brainstorming and discussions will continue, as we travel to GDC this week.

Some of our feedback included the below:

– Design challenges of the Echo?
– Treating Alexa as its own entity is interesting (like Her or Hal)
– How do you know if it’s a success?
– Binary Prime interaction – how do you know if you have enough?
– Narrative changes if the demo includes kids
– Pick up on listeners’ own paranoia (Black Mirror / War Games / 2001 Space)
– Fairly evocative of what kind of story with this interaction – CSI computer
– Using the echo to “find stuff”
– Pay attention to metaphor and user fantasy (wish fulfillment, pleasing, tricks)
– Relationship with Alexa – who is the user in the story?
– speaking as a superpower (lawyers, wizards, detectives, lovers…)
– Be sure to prompt the user to what role they play
– How do we perceive choices given to us verbally?
– Needs personality (Her, Hal, Siri) – “Hey Puter”
– Think about categorization for menu selection
– What will you do with this? What is our case?
– Need to account for a wide variety in “Binary Prime”
– Challenge to make linear elements engaging
Much of our discussion used our short demo as a jumping off point both for the design of our natural interactions as well as the potential of the story moving forward.  We ran through most of the afternoon without issue, but a bug developed where the sound file would abruptly exit the skill in the middle of a session (unprompted).  As a result, we had two groups unable to play through the full demo.

Our demo consisted of a scenario where the user played a judge in deliberations for a bench trial in a case based on Little Red Riding Hood – the script and details are featured in the last blog entry.

After discussing the comments as a group, the team decided to reexamine what we wanted our “discovery” project to be – many little projects that show off interaction, or one prototype that tries to incorporate them all?  Something that has an emphasis on story, or something that uses story as the vehicle to serve the interaction development?  We all knew that story and interaction needed to work hand in hand, but felt a bit challenged when we tried to envision a story to craft in the remaining 10 weeks of the semester.

We returned to the elements of our original pitch – linear story, focus on the method of delivery (smart home) and if we could adapt it from an existing story structure, we would.  Both Seth and Sarabeth spent the weekend going through our inspiration board, focusing on the structure of Orson Welles’ War of the Worlds radio broadcast. What struck us was how it broke the story into two parts, the first a alternate reality of a radio broadcast, and the second, a diary account of a character featured in the first part.
Moving forward, we will be using the structure of WotW as inspiration for how to craft a story that utilizes the circumstances of its own delivery to the user, namely interacting with Alexa as an antagonist.  On a technical note, Phan and Roy have begun playing with new interaction methods, including emotion level and menu selection.  The next week we will come together to outline a combination of design desired interactions, confirmed interactions already developed, and stretch goals for how we can best integrate both sides.

Technical Development:
This week, Roy and Phan worked on developing the technical foundation for the binary choice and the “tool” mode interaction.  To recap, binary choice questions are the first stage in how we are crafting a seemingly open-ended question model, and the “tool” mode is our exploration tool, right now consisting of “start”, “replay” / “start again.”  In addition to this, they’ve developed an intent system which allows us to 1) Make intent, 2) Craft phrases that invoke intent, 3) Use “slots” to replace the related words in phrases, 4) Ultimately craft an “utterance”.  In addition, they discovered through research that we have a 90 second recording limitation to play audio files, meaning that we need the system to prompt the user with a question every 90 seconds.  If it breaks the 90 seconds, the system will shut off the skill, and only the hard coded commands will be active (meaning stop, play etc).

Intent System
Intent A -> [utterance 1, utterance 2…]

• utterance
  • e.g. {Command} this {file} Computer Play
  • Command / file = Slot
• Slot
  • none, list of all the values
  • e.g. command [play, open, start]
• Intent
  • name
  • e.g. computer play

Design Development:
Sarabeth wrote out a script this week based on an interaction that Phan had suggested earlier last week – the judge scenario.  In this, we wanted to primarily test out the binary choice interaction, with a stretch goal of creating the “tool” interaction (as stated above).  We were inspired the game Her Story and the movie The Princess Bride, for different reasons.  In Her Story we took away 1) the player as a character, solving a mystery was powerful in prompting engagement 2) the research / discovery interaction gave the story interaction without delving into branching territory.  For the Princess Bride, we looked at 1) how it was able to frame two stories in one, with a guide character introducing the main plot and 2) How a guide character could help the user navigate in way that didn’t take the user out of the narrative.

In the current narrative draft,  we took a simple story, in this case Little Red Riding Hood, and turned it around to fit our judge scenario.  We did this for a couple reasons

1.  It would allow us to jump right into quickly developing a narrative without the need for much research to build up the world (narrow in focus)
2. Give Seth a springboard for how to start developing an immersive audio environment (music, foley, voice acting)
3. Create a short demo without the need for context for the user, in order to achieve our technical aims (e.g. testing the binary choice interaction)

Seth spent the week working with classmates who could provide voice acting for our scenario, creating music and foley to round out the environment, and conducting research through interactive Alexa stories (Earplay, Wayne Investigation…), podcasts and audiobooks.  In addition to these elements, he developed a sonic logo for our project.

Production:
In anticipation of Quarters, we’ve been working to put together our first technical demo to walk through our first iteration with the faculty.  During the walkarounds, we plan to show a quick slide deck overview of our project, our first demo, and our composition box (first created for the Playtest to Explore Workshop in the second week of school). This week we also finalized our branding materials, which included a logo, halfsheet and poster.  The branding critique provided us with some useful feedback, notably poster layout (more prominent Echo placement) and logo color (less red)

Looking ahead, we will be working on our demo, and taking our findings to further develop a story for the final experience.

This week, Campfire continued to build our technical foundation, and took our playtesting results from last week to better brainstorm what kind of interaction we wanted to create.

Scott and Ralph challenged us at the start of the week to think about what might bring our guests “out of the experience” and strategies to bring them into it.  Our initial “Wizard of Oz” prototype showed us that the Q&A approach needs a tremendous amount of recorded, scripted content, something that might be at odds with our goal of incorporating AI/machine learning.

After some brainstorming about content and thinking about why we pitched this project last semester, we came away with several points.  First, we wanted to create an immersive audio environment.  Second, we wanted to explore how AI and machine learning could integrate into this, using the echo and Alexa.  Third, the narrative could be nonfiction or fiction, and we want to focus on “customization” over “choose your own adventure.”  To that end, we decided to focus our next prototype on the types of interactions that would work with our current technology progress: binary choices and a “remote controller” interaction (skip, fast forward, rewind, replay, etc).  Our narrative for the next technology goal, in focusing on these interactions, includes the guest the central character as a judge, using her/his Computer (Alexa) to review evidence before delivering a bench verdict.  In addition to the guest and Alexa, a scripted character will join the experience as a clerk, driving the story forward.  Over the weekend, we ‘re developing an outline and script, and into the week Seth will work on incorporating these elements into the audio landscape of voice recording and SFX.

Sound Design & Research:

In early development for generating audio to support a narrative experience, we sourced sound design examples from a variety of related media.

• Examining old-time radio dramas (e.g. The Shadow, War of the Worlds, The Hitchhiker’s Guide to the Galaxy, among others) provided examples of how background music, foley, narration and character acting can be timed together and mixed to maximize engagement of their audience.  Audiobook dramatizations tend to follow in this model.
• Narrative podcasts (e.g. The Moth, Serial, 99% Invisible, etc.) seemed often to focus less on the dramatic unveiling of events—thereby deemphasizing their use of sound effects.  Instead, the sound design relies more heavily on ambient and background music curation to provide emotional content and connect scene to scene.

We also reviewed other audio structures and sound design passages.

• Movies, such as “The Princess Bride,” had been a tossed-around reference for a while for its creative combination of narration and character-acted storytelling.  Other examples from directors like David Lynch, Terrence Malick and Sergio Leone show the ability of audio to create tension and make use of indirect control.
• Storytelling and concept-based musical albums (e.g. Harry Nilsson’s “The Point“, The Flaming Lips’ “Yoshimi Battles the Pink Robots“, and others) show how to create a musical moment within the story, should that be a path we choose to take.
• Selections from ambient recording artists, like Aphex Twin, provided inspiration of the kinds of music that can support, but not overpower, voice recordings.
• Finally, we began reviewing sound design elements from sound libraries: Sonniss and Spitfire Audio.

Technology:

For the technology side, Roy worked on testing different features of IBM Watson to make a chatbot, using the AlchemyLanguage, Natural Language Classifier and Retrieve and Rank APIs.  He tested out the possibility of using them in our own AI later on, which helps for classification of user intent and retrieving information to pass on to them.  Roy and Phan started implementing the front-end side of a basic Alexa Skill prototype, for setting up basic interactions for our next prototype using Alexa (the judge narrative).  In addition, Phan spent time testing the AI’s ability to respond to binary questions, focusing on how it would perform answering vague or incomplete phrases, as well as setting up our project’s work on the ETC server from his initial setup on the Heroku.  Phan’s development into the Alexa framework (how to communicate with Alexa) and the basic framework for the story tool will allow Seth and Sarabeth to further implement the design and interaction of the narrative prototypes in the coming weeks.

Next Week:

In addition to work on the next prototype, Seth created a draft of our branding materials, including logo and poster design.  We took our team picture, featured above, (with photoshop assistance from Yen).  We also skyped with our classmates in SV (Sweet Talk) who are working on a VR experience using the Vive and Alexa voice commands.

Looking ahead, the team will continue to work towards our next prototype goal in anticipation of 1/4 Review, finish up branding work, and schedule a Skype conference with Rich Hilleman (Amazon) to discuss our progress to date.