Usability Test - LLM Travel Assistant





How well does this chatbot implementation serve my specific query?

Scope: Plan a honeymoon
-I don’t have a clear idea.
-I need flights and hotel.
-I want to stay near the beach
-I have $5000 budget
-6 day trip.

This is the persona background and the “Job’s to Be Done”. 
• Part of our wedding planning processes, I was tasked with making honeymoon arrangements. I need to demonstrate how much my partner means to me with some well thought travel options. 

Emotions: Excited, Stress, Curious, Ambitious, New Experience.

Here are my screenshots as I worked through this scenario and captions of the notes I took during my experience.







My first attempt at “zero-shot” prompt had mixed results. There were features in unexpected places in the UI and it didn’t return hotel options nor international destinations. There was also not enough UX writing to support using this tool. I found myself guessing and unconfident about where to look to address some of these issues.

I wanted to see how the results would change if I added a Travel Agent role to my prompt. The character limit of the prompt would be a constraint but I worked within it.

 




Including a role definitely improved this model’s performance though it didn’t capture all of my search requests, notably the total price of the two airline tickets.

More unexpectedly, when I removed the the role in my prompt after looking at the site’s list of suggested prompts, the return seemed to break the UI and returned a Chat-GPT style unstructured list of options.





How might we improve the interface of AI search experience to improve user adoption?

The issue I found with this interface is that I felt that my success is dependent on my prompt engineering ability. Is having a cursory understanding of prompt engineering a reasonable expectation of someone looking to book a trip in 2025?





The template prompts are good starting place but there is potential for improvement. What would it look like if enterprises provided customers with a Madlib-style prompt construction that they could fill with their trip details?


  • Reducing user error conserves computing resources by removing unskillful prompting.

  • Playful interface reduces friction and barrier to entry.

  • Reduces cognitive load on customers, enable them focus on their immediate goals.








+1 503 680 0674
stern.aaron.b@gmail.com

“Thanks for stopping by!”