Main navigation and search bars on the NYPhil's digital archives homepage.
Purpose: In 2020, the New York Philharmonic's Shelby White & Leon Levy Digital Archives, the “oldest and most comprehensive collection of any symphony orchestra,” received a grant from the National Endowment for the Humanities to optimize, expand, and ensure the sustainability of their website to better serve their users. As part of this work, the Digital Archives partnered with the Pratt Institute Center for Digital Experiences in fall 2022 to improve the discoverability and accessibility of their archival materials for a broad audience including researchers, music students, musicologists, historians, genealogists, and casual visitors. Our team of four UX consultants self-selected into a team that would work with them to accomplish that goal through a series of moderated user tests and a corresponding usability evaluation.
Goals: Gain experience working with UX clients as a UX Consultant with the Pratt DX Center. Improve the exploratory experience for new users landing on the NY Phil's Digital Archives site, plus evaluate and improve the experience for experts and researchers seeking out specific archival information.
Results: A Moderated User Testing Study, Evaluation Report, and Case Study (below).
Based on analysis of our user tests, the final report presented five major findings and recommendations for the Digital Archives' consideration:
#1: Improve navigation usability and accessibility by updating styling, spacing, and text.
#2: Update style and navigation elements on the carousel to make it more intuitive and accessible.
#3: Streamline search results page with consolidated sort and filter options and highlighted search results.
#4: Move pop-up modal underneath each filter and introduce a search bar for the filter options when opened.
#5: Other minor findings and recommendations.
My Role: Acted as a UX Consultant with the Pratt DX Center as part of a team of four other consultants and graduate assistants. During recruitment and client management, I lead the sending of initial recruitment emails with consent information to potential participants, set up the scheduling system for the team, and stayed in email contact with our client representative at the Digital Archives. In drafting the report, I wrote one of the recommendation sections, the methodology section, and parts of the appendix, as well as took part in editing of the final full report.
Methods: Co-building strategy with clients; Recruitment of target users; Conducting moderated user testing sessions; Quantitative assessment via System Usability Scale (SUS) survey; Compiling and synthesizing qualitative findings with the team via Rainbow Sheet method.
Read more about the methodology in the team's full report.
Skills: Moderated User Testing, UX Research and Evaluation, Survey methodology, Team Management
Tools Used: Figma; Zoom; Google Sheets, Forms, Slides & Appointments
Learning Outcomes: Communication, User-centered Design, Critical Perspectives
Case Study: Streamlining Access to the New York Philharmonic's Digital Archives
Strategizing with the Client to Study & Improve User Experience
At the kickoff meeting, our team asked exploratory questions to get to know the Digital Archives’ mission, target audience, and usability questions and concerns, then set out to co-create the usability study’s scope with our client. He took us through an overview of the site’s history and two reasons for a redesign: optimizing usability and accessibility for potential and existing users. We learned that existing users are likely to be a combination of a) researchers regularly performing investigative searches, and b) non-scholarly users performing casual research — for instance, seeking records on a relative, a specific performance they attended, or a piece of music they enjoy.
From this, we identified two guiding strategic goals: 1) Improve the exploratory experience for new users landing on the site, and 2) Evaluate & improve the experience for researchers seeking out specific archival information.
Recruiting Target Users: “Amateurs,” “Explorers,” & “Professors”
Following our meeting, we identified these three target user groups: 1) Amateurs: Students and other first time visitors to the digital archives site. Highly familiar with the internet and search tools. Maybe looking for a specific resource, e.g. a piece of music or photograph of a relative. 2) Explorers: Young professionals with some exposure to the New York Philharmonic. Exploring what the archives have to offer. 3) Professors: Researchers highly familiar with online archives and the Digital Archives site. Seeking deep dive research into a subject or person.
We aimed for six to eight users total, with two from each group, especially targeting at least two from the Amateurs and Professors groups to reach both the “new user” and “expert/researcher” audience. We launched a Google Form to collect users interested in participating, as well as to determine what, if any, user group they fit into. Our team circulated the form on both the Pratt listserv and, through our client, with visiting student groups and other internal NY Phil representatives.
My Role: Structuring Team Responsibilities on a Tight Timeline
Recognizing early on that scheduling a slew of user tests via email could become hectic, I created a new scheduling system through Google Appointments to streamline user test logistics. To do this effectively, first our team shared with each other our hourly availability. From there, I created a Google sheet displaying the times over the following two weeks that would include two moderators for each time (one leading and the other note-taking). I then created a Google appointment calendar for the potential participants to schedule their preferred time for a user test session, as well as initial and follow-up scheduling emails that contained everything that moderators and participants would need to know in advance of user test sessions. From there, my teammates compiled everything into a recruitment guide for everyone to use, and we split off to tackle moderating at least two recorded user test sessions each.
Working with three fellow Pratt School of Information graduate students, our team strengths from the beginning lay in our ability to look at the full scope of what we need to accomplish, then to divide up tasks between different team leads. For instance, my teammates drafted the script and the other required documents for user testing, then we split off to use all of these as moderator leads and note-takers in the tests. Once it came time to write the report, we all put our notes into a consolidated Google sheet, then met together to discuss and analyze major findings. Finally, we divided up sections of the report to write, leaving extra time before each deadline for everyone to review and comment on each other’s writing.
Matching User Test Methodology to End Goals
A remote moderated user testing approach matched our project scope and goals well: it allowed us to obtain substantive results with relatively few users (it required at least five), to easily screen and voice record the tests over Zoom, and to cast a wide net for participants during recruitment, regardless of their location. It also allowed us greater flexibility in scheduling for both users and ourselves. Each moderated user testing session involved two representatives from the team, with one facilitating the test and the other note-taking, together leading users through a series of tasks focused on the big user goals as identified with our client. See right for the full task list with user goals.
We conducted a total of ten remote moderated user tests (surpassing our goal!) on Zoom in just under two weeks. The usability study included a diverse set of participants, ranging from those with expertise with the Digital Archives to users with little to no familiarity with the Philharmonic or its archives. Seven we identified as “Amateurs” or “Explorers,” while three fell into the “Professors” category.
After the tasks ended, participants took a brief System Usability Scale (SUS) survey and answered one emotion-focused question to gauge their overall experience. Once all tests had wrapped up, the team reconvened to compile and synthesize findings via the Rainbow Sheet method, prioritize the findings by severity, and create a handful of actionable recommendations addressing the major issues this list revealed.
Complimentary First Impressions, Challenges with Finding Information
Overall, first impressions of the Digital Archive were positive. Users appreciated the clear Search feature on the homepage, the welcoming feel of the page in comparison to other archives sites, and the variety of types of resources made available to them. Several also noted how interesting it was to see the markups included on digitized documents.
We also asked users at the end of their tests for some qualitative feedback to describe their experience. As can be seen reflected in this word cloud, it was mixed. Users identified an average of 8.4 problems each across the home page, search results page, and other areas. Words they used to describe their experience ranged from more neutral (coherent, contemporary) to positive (attractive) to negative (inefficient).
We asked users to complete a brief quantitative System Usability Scale (SUS) survey for the NY Phil to use as a benchmark for testing against any improvements they made to the site, based on our feedback.
The site initially received a below-average score of 60.3, leaving it with a just an “OK” measure of usability. Those detailed results, and how to evaluate them, were discussed in more detail both with the client in our presentation and in our full report.
Recommended Redesigns for Home & Search: We identified five general areas for improvement, but essentially four major recommendations across both the home page and the search results page. All recommendations focused on improving site usability through accessibility updates, layout tweaks, consolidating features and more. The 5th recommendation included miscellaneous additional findings and areas for further investigation.
#1: Improve navigation usability and accessibility by updating styling, spacing, and text.
#2: Update style and navigation elements on the carousel to make it more intuitive and accessible.
#3: Streamline search results page with consolidated sort and filter options and highlighted search results.
#4: Move pop-up modal underneath each filter and introduce a search bar for the filter options when opened.
#5: Other minor findings and recommendations.
Home Page:
On the home page, several users experienced issues using the main navigation bar and the carousel. One user echoed the experience of several: “It’s luck to see what the page loads on the carousel and if it wouldn’t have showed up [via the automatic timing that flips through the banners], I wouldn’t have known that it was there.”
Recommendation 1: Improve navigation usability and accessibility by updating styling, spacing, and text.
This included reducing logo size and white space, changing “Features” to “Exhibits,” and adding styling to the navigation elements to indicate the current page to the user.
Top: Before // Bottom: After (With recommendations implemented)
Recommendation 2: Update style and navigation elements on the carousel to make it more intuitive and accessible.
Includes changing the navigation and Explore button colors and un-hiding navigation arrows.
Top: Before (Current carousel image) // Bottom: After (With recommendations implemented)
Search Results Page:
As users tried to find and narrow their search results through various filters, they got stuck in different areas of the results page, either due to missing or inconsistent content or a lack of direction with how to navigate the many different filtering options.
Left: Before recommendations (Current search results page)
Recommendation 3: Streamline search results page with consolidated sort and filter options.
This includes renaming the lefthand section to “Filter,” consolidating the top navigation filters into the lefthand section, consolidating and enlarging the “Sort by” options, and displaying search results with thumbnails in a layout consistent to all other resource type listings.
Right: After (With recommendations implemented)
The existing lefthand filters were also frequently overwhelming for users, who found themselves sometimes having to navigate through 20+ pages of names to find a particular composer or musician’s name.
Recommendation 4: Move pop-up modal underneath each filter and introduce a search bar for the filter options when opened.
Left: After (With recommendations implemented)
Digging Deeper into Metadata, Accessibility, & Site Evaluation:
One major challenge that came up in my tests was users getting stuck trying to use the “right” keywords to find information. One of the users in one of my sessions said, after trying a search several times with no satisfying results, “When you’re searching and you get a lot of zeroes, you know you’re using the wrong terms.” Another noted that they “had to use the trial and error method to find this [information].” We saw this crop up a few times, wherein searches that users expected to yield results yielded no results — or even incorrect ones. One user received zero results when they searched “Tchaikovsky’s nutcracker ballet,” then tried to correct that by searching instead for “Tchaikovsky’s nutcracker,” and still found no relevant results.
In our fifth recommendation section, we decided to recommend further investigation of the backend information organization and metadata to address these inconsistencies in search results. Additionally, team members looked into the site’s layout and formatting with accessibility best practices in mind and included in the final section a shortlist of relevant recommendations. One example is our recommendation to change all of the website’s “–brand-blue” elements to “–brand-blue-ada’” in the CSS styling.
Among other minor recommendations included in the fifth section, we also recommended that the Digital Archives run further SUS surveys and facilitate additional moderated user testing again once they implement any future site improvements, in order to test their effectiveness.
Revealing Recommendations to the Digital Archives:
We presented these findings and recommendations in detail to the Digital Archives team, and were pleasantly surprised to find that they not only wanted to hear about the “low hanging fruit” of recommendations, but also were interested in “pie in the sky” visionary ideas for improving their site. Hearing that, especially regarding our carousel recommendations, we revisited our report and included more “inspiration” website examples for them to consider from both The Met (with a different layout alternative to using a carousel on the home page) and the Museum of Science & Industry (featuring a carousel with more feedback and accessibility features included).
We closed out the project by inviting the Digital Archives team to get in touch with any questions once they had reviewed the full report, and reiterated the positive experience we had getting to know their website and the Philharmonic’s rich history.
To view our final client materials, click the following links for the client presentation and full moderated user testing report.