CS Lecture with Rohit Prasad: Vice President and Head Scientist Amazon Alexa
The following day, the group attended the SCS Distinguished Lecture featuring Rohit Prasad, an executive and developer on the Alexa team with Amazon. We were heartened to hear that the focus areas from Amazon were similar to our efforts during the semester, namely their effort to bring conversational context (this open-ended interaction) to Alexa interactions overall. Our main takeaway from this lecture was the fact that the technology isn’t there yet, but isn’t that far away. Perhaps in 5 years, conversational chatbots might be more in reach. In the present, Amazon released Amazon Lex, a way for developers to start experimenting with these very challenges.

Week 16:
This week, we spent preparing our final presentation, as well as our deliverables for the semester’s end. During this time, Amazon announced Echo Show, a new hardware for the Echo system that incorporates a visual screen as well as added utility functions: messaging. This was another element that validated our efforts: messaging is a meaningful and gratifying addition to this platform. Roy’s asynchronous messaging system were in line with Amazon’s direction, as our open ended experience was.  Indeed, the night before our presentation, Amazon deployed a software update to the Alexa app that allowed for voice calls and messages between Echo devices. While we worked to create an entirely audio UX for this space, Amazon relied on its own app, and interfaced with users’ mobile phones.

Final Presentation:
For the final presentation, we chose to focus on our efforts to explore the interactions within the space, and discuss it within the context of Amazon’s current traction and future development. Some of the questions from the audience related to the future of voice recognition in this space, and our team agrees – once voice recognition becomes conversational enough (and aware of conversational context), it will play a large role in UX design, from audio only projects, to VR and more.

Final Play Through:
In the final play throughs, we talked a bit more in depth about our findings overall, and ran through some of the demos we hadn’t shown off to the faculty. They urged us not to let our work disappear, and to this end, we’ve been adding even more information to our handbook, are creating videos showing play throughs of all our demos and plan to create a short article for publication online.

Final Deliverables:
1. Developer’s handbook
2. Alexa Skill Package
– Intruder Alert (Query)
– HaKoo (Syllable Recognition)
– Puns with Friends (Asynchronous Messaging)
– Fortune Teller (Template)

1) Story Demo (Open Ended Questions)
After voice recordings in week 12, Seth spent the majority of his time cutting the clips and creating a track of the entire experience. His goal was to find the story flow, and integrate appropriate SFX and music to accompany the experience. The main challenge was to use music and SFX to elevate the emotions of the story, particularly difficult in a 5 minute experience. In addition, Seth needed to consider the breaks in sound (every time a user interacted) when he composed the music to the experience. He settled for an ebb-and-flow style that would make the breaks for interaction less jarring for the user.

For the story, Sarabeth worked on expanding the number of possible answers in the “open ended” section with our voice actor, and tested throughout with the query system in the editor tool. Now the query system, contrasted with the intent system, allows us to parse sentences and assign it a probability to a given answer. Training it consists of entering in keywords, sentences and other inputs to help it make more accurate assessments. In addition to expanding the depth of the script (for the open ended section), Phan and Roy added the features of transcript retrieval and keyword function. We wanted to add this feature for several reasons. First, we found through playtesting that the first interaction with the user sets the stage for the rest of the experience. For example, having a yes-no question work with “sure” or “definitely” or “Ok” instead of only “Yes” frees up the possibility of nuanced responses. We wanted to set the tone that Alexa was really listening to the user, and decided to try out having her repeat the user’s answer back to them, and classifying it in a category from there. The second choice was including the user’s name. This allows us to immerse the user a little more later on in the story, letting Alexa address him/her by name.

2) Ha Koo (Patterns) Demo
Phan and Seth worked closely to develop the sound and story to support the pattern structure. They decided to proceed with a narrative demo for this interaction for a couple reasons. First, it provided a contrast with the main story demo in giving our guests a light-hearted option. Second, it demonstrated in very basic terms the recognition of syllables (contrasted with the word-based query recognition in the interactions above). The structure of the demo focused on gradually ramping up understanding and teaching of the “ha koo” langauge for users, and ends with the users navigating a conversation entirely with phrases “ha koo” and “koo koo”.

3) Messaging Demo
Roy’s asynchronous messaging demo integrated into a puzzle game, complete with SFX. Originally, we had planned to show this during the Softs walkaround, but were advised to leave room at the end for questions with the faculty.

Our Softs Feedback:
-show all demos to help show range of possible interactions with platform
-explain the limitations of the platform
-make a deliverable that you can share with the dev community
-enter work in the amazon echo competition

For finals, we plan to deliver the package of demos (4 in total) showing off different interaction types. (Template, Navigation, Query/Open-Ended, Pattern, Asynchronous Messaging). In addition to this, we will be creating an Echo developer handbook. Leading up to the ETC Showcase, we will be making revisions to the demos, adding user reprompting (so ensuring that there is enough time for users to respond) and packaging all the demos together via a navigation interaction.

Looking ahead we will be completing our documentation, filming our promotional videos and compiling our records for the end of the semester.

• Need more range in the “open ended” questions when guests speak with our voice acted character
• Shorter answers to the open ended sections, but more pointed to guest questions (for us this means crafting more options for what we anticipate guests will ask)
• First interaction sets the tone for the experience – if Alexa responds appropriately to “Sure” or “I guess” the same as “Yes” we’ve opened the door beyond “yes” or “no”

In addition, the team worked on the short demos for multiplayer messaging, pattern recognition as well as a template answer demo. As a reminder, our goal for the semester to deliver 3-4 small demos, each demonstrating different interactions. Intro – Navigation to demo, 1 – Story, Open-Ended, 2 – Pattern Recognition, language teaching, 3- Asynchronous Messaging Game, Closing – Message to developer. The interactions we do not include will be part of our documentation at the end, showing our design and technical efforts for the semester.

On Friday, Seth brought in 3 drama students from main campus to do voice recording for our story experience (open ended) and the Pattern Recognition (hereafter referred to as “Ha Koo” – the alien language it teaches users). We found their talent really elevated the sound experience of the story. Over the weekend, Seth worked on cutting the clips, and we will plug them into the editor to continue development. Seth composed background music, and we will be looking to secure musician talent next week (perhaps Roy will double as our violinist). Seth’s challenge in composing is twofold: first, something that will offer dramatic richness to the soundscape. Second, something that will work in subtle phrasing, that allows for a logical flow between periods with no sound (user input sections – ie. when the user is asking a question) and sound files.

In Week 13, we will be working on our 4th story demo (voice recording) as well as finalizing our demo package to show at softs.

For our project deliverable, we’ve decided as a group to focus the story on the “open ended” interaction, and provide short demos to showcase the other interactions we’ve developed. What that means for our project delivery is the following:

Project Deliverable:
Campfire Skill:
Navigation to demos (Showcasing our Navigation interaction)

• Main experience demo (Story+ Open-Ended interactions) – More polish than the other demos (voice acting, SFX, music)
• Short demo showcasing Template answer interaction
• Short demo showcasing Pattern-based Voice Recognition
• Short demo showcasing Asynchronous Messaging between Echo devices

In addition to our demo package, we will be documenting our interaction development, design decisions and discoveries in a report that will hope to help future developers in creating new and natural ways for interaction with Smart Assistant devices, as well as design discoveries for audio-only experiences.

Below you’ll find a list of the events with our insights, as well as some of our planned work moving forward.

Week 10
Small Emotion Playtest (Thursday at HCI Playtest Lab) – 4 testers

• Test used 1 dialog line, read with different inflections / emotions
• Discovered that strong emotional responses prompt more user engagement (ie
  , the more rude or outrageous the recording was, the more likely the user would respond)
• Nuanced emotional readings difficult to gauge for listeners

ETC Playtest Day (Saturday at ETC) – 17 testers

• Story length should be 3-5 minutes, not 7-10 minutes (previous)
• Any period longer than 15 seconds of no interaction loses user engagement
• The story topic pertinent
• “Alexa” wake word interrupts skillset, skips through experience
• Open-Ended interaction moment worked for most users
• We also tested the pattern-based voice interaction – users responded positively to a shift in content
• Potential to “learn” a new language (in pattern format) felt fun, and exploration felt fun
• Users didn’t need a “ramping up” of interactions – best to jump right in
• Past experience with the Echo didn’t really affect the results we had

Story Demo

• Recorded with scratch recording from classmates
• Initial structure tested at Playtest Day had combination of passive listening and interaction
• Launched to Echo via editor tool and server
• Further exploration with sound effects, integrating different voice acting as well as other “robotic” voices to sit with Alexa in the experience

Editor Tool Development

• Editor tool revised to include favorites functionality, sound file, intent training and story branching / fail case mapping.
• Able to export directly from tool for live testing
• Some content bugs led to crashing, later corrected in subsequent updates
• Through testing, discovered that 90 second “limitation” has a work around (if you separate distinct sound files form the editor node), but this ended up being a moot point because user attention waned significantly after 15 seconds of passive listening

Asynchronous Messaging Development

• Basic functionality of messaging implemented
• Result: user dictates message, and recipient details. Retrieved message read by Alexa.

Week 11
Story Iteration

• Based on feedback from the playtest day, we shortened the structure to create a 3-5 minute experience, with interactions from the start
• Based on conversations with Anthony Daniels, renewed focus on interactions balanced with performance
• Story read through with Ralph and Scott provided good feedback on tightening up the dialog, structure as well as how to frame the interactions, both open ended and template based

Technical Developments

• Further editor tool development, with added functionality in “open-ended” training (query intent), version control download and upload files, bug fixes and UI adjustments and “bubble set” arrays that allow for varied outputs (so for failcases, we can cycle through different reactions).
• Updated asynchronous messaging system to have the ability to store multiple messages from a single user, as well as add sound emojis to the end of your messages.

Voice Acting

• Seth reached out to contacts on main campus drama dept, received a connection to voice acting talent from the CMU acting class
• Reaching out to voice actors next week, have 9 potential candidates

Alexa Developers Open Hours

• live Q&A for technical discussions and development
• Roy participated, and his notes are here
• One notable discovery: we’ve been having some challenges with Echo shutting off mid-skill. Since we’re able to track that the server wasn’t the source of the issue, the Alexa dev team suggested it might be caused by inconsistent internet connection.

Looking ahead, we’re recording a revised scratch demo for further story testing, and plan to coordinate with voice acting by the end of the week.

This week, Campfire continued to improve our intent system, editor tool and develop further on the asynchronous messaging skill and the nonverbal pattern game skill (think of it as an alien communication game). On the design side, we worked on recording scratch voice acting (recordings will run into Monday) to try and fold in our story draft into Phan’s editor tool. After meeting with Brenda this week, we also recorded assets to try and develop a targeted playtest for how users respond to Alexa when she says the same sentence, but in different emotional tones (think flirtatiously, menacingly, etc).

Short list from the week:

Half Presentation (Video Below)
Twine Story Draft
Scratch Recording – into the weekend
Editor tool updates, intent system updates
Demo using the editor tool

Looking ahead:

Story entered into online editor tool
Scratch recording of story, to test on Echo
Alexa tonal playtest (to test user responses to Alexa’s voice in different emotional tones)
Interaction refinement for in progress interactions (e.g. asynchronous messaging)
Story updates
Training open ended responses
Scheduling for final voice recording
Playtest to Refine Workshop

Before departing for GDC, the team worked on settling up a story and game plan for the time leading up to our Halves presentation. Story-wise, we’ve settled on creating an interaction with War of the Worlds (Radio broadcast) as an inspiration. Sarabeth met with Chris Klug before departing, to talk through some story structure questions, and after a draft of the story is complete, planned a table reading with both Ralph and Scott the following week.

The week after GDC, Seth and Sarabeth fell prey to a conference illness, which derailed some development for the week.

Phan and Roy continued technical development, updating and improving the intent system, as well as creating a story editor for Seth and Sarabeth to use as we start to integrate the design elements into our technical foundation.

This week during Spring Break, the team will be meeting to go over the first draft of the story, create our presentation for next week, and meet with Scott to give a project update.

Technical Development:
This week, Roy and Phan worked on developing the technical foundation for the binary choice and the “tool” mode interaction.  To recap, binary choice questions are the first stage in how we are crafting a seemingly open-ended question model, and the “tool” mode is our exploration tool, right now consisting of “start”, “replay” / “start again.”  In addition to this, they’ve developed an intent system which allows us to 1) Make intent, 2) Craft phrases that invoke intent, 3) Use “slots” to replace the related words in phrases, 4) Ultimately craft an “utterance”.  In addition, they discovered through research that we have a 90 second recording limitation to play audio files, meaning that we need the system to prompt the user with a question every 90 seconds.  If it breaks the 90 seconds, the system will shut off the skill, and only the hard coded commands will be active (meaning stop, play etc).

Intent System
Intent A -> [utterance 1, utterance 2…]

• utterance
  • e.g. {Command} this {file} Computer Play
  • Command / file = Slot
• Slot
  • none, list of all the values
  • e.g. command [play, open, start]
• Intent
  • name
  • e.g. computer play

Design Development:
Sarabeth wrote out a script this week based on an interaction that Phan had suggested earlier last week – the judge scenario.  In this, we wanted to primarily test out the binary choice interaction, with a stretch goal of creating the “tool” interaction (as stated above).  We were inspired the game Her Story and the movie The Princess Bride, for different reasons.  In Her Story we took away 1) the player as a character, solving a mystery was powerful in prompting engagement 2) the research / discovery interaction gave the story interaction without delving into branching territory.  For the Princess Bride, we looked at 1) how it was able to frame two stories in one, with a guide character introducing the main plot and 2) How a guide character could help the user navigate in way that didn’t take the user out of the narrative.

In the current narrative draft,  we took a simple story, in this case Little Red Riding Hood, and turned it around to fit our judge scenario.  We did this for a couple reasons

1.  It would allow us to jump right into quickly developing a narrative without the need for much research to build up the world (narrow in focus)
2. Give Seth a springboard for how to start developing an immersive audio environment (music, foley, voice acting)
3. Create a short demo without the need for context for the user, in order to achieve our technical aims (e.g. testing the binary choice interaction)

Seth spent the week working with classmates who could provide voice acting for our scenario, creating music and foley to round out the environment, and conducting research through interactive Alexa stories (Earplay, Wayne Investigation…), podcasts and audiobooks.  In addition to these elements, he developed a sonic logo for our project.

Production:
In anticipation of Quarters, we’ve been working to put together our first technical demo to walk through our first iteration with the faculty.  During the walkarounds, we plan to show a quick slide deck overview of our project, our first demo, and our composition box (first created for the Playtest to Explore Workshop in the second week of school). This week we also finalized our branding materials, which included a logo, halfsheet and poster.  The branding critique provided us with some useful feedback, notably poster layout (more prominent Echo placement) and logo color (less red)

Looking ahead, we will be working on our demo, and taking our findings to further develop a story for the final experience.

This week, Campfire continued to build our technical foundation, and took our playtesting results from last week to better brainstorm what kind of interaction we wanted to create.

Scott and Ralph challenged us at the start of the week to think about what might bring our guests “out of the experience” and strategies to bring them into it.  Our initial “Wizard of Oz” prototype showed us that the Q&A approach needs a tremendous amount of recorded, scripted content, something that might be at odds with our goal of incorporating AI/machine learning.

After some brainstorming about content and thinking about why we pitched this project last semester, we came away with several points.  First, we wanted to create an immersive audio environment.  Second, we wanted to explore how AI and machine learning could integrate into this, using the echo and Alexa.  Third, the narrative could be nonfiction or fiction, and we want to focus on “customization” over “choose your own adventure.”  To that end, we decided to focus our next prototype on the types of interactions that would work with our current technology progress: binary choices and a “remote controller” interaction (skip, fast forward, rewind, replay, etc).  Our narrative for the next technology goal, in focusing on these interactions, includes the guest the central character as a judge, using her/his Computer (Alexa) to review evidence before delivering a bench verdict.  In addition to the guest and Alexa, a scripted character will join the experience as a clerk, driving the story forward.  Over the weekend, we ‘re developing an outline and script, and into the week Seth will work on incorporating these elements into the audio landscape of voice recording and SFX.

Sound Design & Research:

In early development for generating audio to support a narrative experience, we sourced sound design examples from a variety of related media.

• Examining old-time radio dramas (e.g. The Shadow, War of the Worlds, The Hitchhiker’s Guide to the Galaxy, among others) provided examples of how background music, foley, narration and character acting can be timed together and mixed to maximize engagement of their audience.  Audiobook dramatizations tend to follow in this model.
• Narrative podcasts (e.g. The Moth, Serial, 99% Invisible, etc.) seemed often to focus less on the dramatic unveiling of events—thereby deemphasizing their use of sound effects.  Instead, the sound design relies more heavily on ambient and background music curation to provide emotional content and connect scene to scene.

We also reviewed other audio structures and sound design passages.

• Movies, such as “The Princess Bride,” had been a tossed-around reference for a while for its creative combination of narration and character-acted storytelling.  Other examples from directors like David Lynch, Terrence Malick and Sergio Leone show the ability of audio to create tension and make use of indirect control.
• Storytelling and concept-based musical albums (e.g. Harry Nilsson’s “The Point“, The Flaming Lips’ “Yoshimi Battles the Pink Robots“, and others) show how to create a musical moment within the story, should that be a path we choose to take.
• Selections from ambient recording artists, like Aphex Twin, provided inspiration of the kinds of music that can support, but not overpower, voice recordings.
• Finally, we began reviewing sound design elements from sound libraries: Sonniss and Spitfire Audio.

Technology:

For the technology side, Roy worked on testing different features of IBM Watson to make a chatbot, using the AlchemyLanguage, Natural Language Classifier and Retrieve and Rank APIs.  He tested out the possibility of using them in our own AI later on, which helps for classification of user intent and retrieving information to pass on to them.  Roy and Phan started implementing the front-end side of a basic Alexa Skill prototype, for setting up basic interactions for our next prototype using Alexa (the judge narrative).  In addition, Phan spent time testing the AI’s ability to respond to binary questions, focusing on how it would perform answering vague or incomplete phrases, as well as setting up our project’s work on the ETC server from his initial setup on the Heroku.  Phan’s development into the Alexa framework (how to communicate with Alexa) and the basic framework for the story tool will allow Seth and Sarabeth to further implement the design and interaction of the narrative prototypes in the coming weeks.

Next Week:

In addition to work on the next prototype, Seth created a draft of our branding materials, including logo and poster design.  We took our team picture, featured above, (with photoshop assistance from Yen).  We also skyped with our classmates in SV (Sweet Talk) who are working on a VR experience using the Vive and Alexa voice commands.

Looking ahead, the team will continue to work towards our next prototype goal in anticipation of 1/4 Review, finish up branding work, and schedule a Skype conference with Rich Hilleman (Amazon) to discuss our progress to date.

On the technology side, Roy and Phan started by putting together a high-level architecture on how the AI would accept user input, and output a response.  This involves taking in an input, and finding keywords, and finding intent from it in an initial encoding phase.  A hidden phase would next take these values from an encoding phase, and generate answer keywords and intent from the story to pass on to a decoding phase.  The decoding phase would initiate a query-based response system and dynamically generated responses on a neural network, to test the effectiveness of both.

In addition, Roy tested out various machine learning libraries and APIs: Theano, TensorFlow, Keras and Chainer.  He and Phan decided to move ahead with TensorFlow + Keras.  Roy also spent time this week researching papers and learning more about the capabilities of RNNs and LSTMs, for developing language models, testing out a simple LSTM in generating text based on keyword inputs.

For the design side, Phan developed a “Wizard of Oz” Alexa setup for Sarabeth and Seth to test initial designs, in order to quickly prototype interaction cases.  Seth and Sarabeth spent the first week investigating story types, examples and scenarios, e.g. short stories, poems, fables, news, politics and sports.  They settled on starting with a recap interaction of the Pittsburgh Steelers AFC Championship game against the New England Patriots, which tragically (for Steelers nation) ended in defeat.  Their strategy for week 2 involved creating a library of information to pull from during a test with a few football fans in the ETC building.  The goal of this first playtest was to see how open ended questions, option based questions, prompts, responses, suggestions, trivia and fail cases affect the flow of conversation.  Though we ultimately want to develop an experience using a linear story, we started with a sports topic to start identifying what common threads we see in audio interactions.  

With our first playtest (Thank you Cody (Pats fan), Mike (Bills fan) and Erika (not a fan)) we have several takeaways:

• Users expect Alexa to know and readily provide stats or fact heavy questions
  • For this interaction, we need to have a way to lengthen the conversation past the transaction
• Surprising moments that play off Alexa’s dry tone, or the contradiction of unexpected fact retrieval give the most rewarding moments of humor
  • “Tom Brady is the GOAT” – Alexa retrieves a commentator’s opinion, and delivers with same tone as a “fact”
  • Question about the Steeler’s hotel fire alarm incident, Alexa responds with a quote from the perpetrator “I got drunk and did something stupid.” (not immediately clear at start of the sentence – surprise at the connection at the end
  • Below are some clips from our playtest:

  
  

• Canned jokes are not as effective as above examples because they are not based on humor from the interaction itself
• Silences are awkward, esp in a group, maybe not a problem solo
• Clips are too long, need to be shorter (max 15-20 sec)
• Context of the setup creates a certain mood
• Connection based on a “shared understanding” – what “baggage” does the user bring to the interaction, and how might this affect the experience?
  • E.G. Venting session vs Celebration of the game
• Found that questions like “who were you rooting for” allowed Alexa the opportunity to join a side- creates a connection to the user, and allows for more curated conversation
  • This goes into “branching” territory

After going through our notes from the playtest, Phan noted that we should focus on how to reduce the complexity and vastness of answer, by creating an environment where Alexa may teach the guest how to better interact with her, integrated into the main experince.  From here, we would expand the experience (because our goal is to have the interaction with users be as natural as possible).

• For our purposes begin by teaching computer-friendly keywords to identify tasks and simplify / specify interactions.
• Once we understand the process, begin nuancing the language to be more human-friendly.

In addition to the playtest, Seth spent time investigating other story-based Alexa “skills” currently available (Ear Play, Wayne Investigations, The Magic Door).  He also started researching what kinds of audio we might integrate, in particular thinking about the kind of delay that will exist between the guest and Alexa interaction – what audio can gracefully fill this space?

For project production needs, Campfire began developing branding concepts for next week’s branding walkarounds, as well as team processes (core hours, Scrum, stretch goals).  Sarabeth attended the Playtesting for Explore workshop earlier this past week, and shared with her team the new project requirements for 1/4s, which included brainstorming techniques, composition box, and strategies for initial testing.  Campfire worked to fill out our project Metrics Matrix, keeping in mind our project goals of design discovery as well as technical prototyping.  

Looking ahead this week, the team plans to define our next prototyping goals (both design and technical) as well as complete our branding materials.