top of page

Generative AI

Create an effective UX research strategy that promotes innovation

Copertina Quality AI.jpg
CONTEXT

Quality is a code analysis tool, developed by Codacy, that assists developers in detecting and resolving issues in their codebase.

During a one-week-long hackathon, a Codacy team introduced a new feature to test Generative AI within the Quality product. The objective was to assess users' interest in this feature and how it can solve their known problems.

PROBLEM

Quality AI provides suggestions in the git provider on how to solve issues in the codebase and support an auto-fix feature. However, we are assuming that suggestions are reliable, compile and can be merged automatically in the PR. 

Additionally, it's worth considering whether these suggestions solve the problem in the most effective way possible.

SOLUTION

Set up a discovery process that helps us understand how we can make an AI feature more valuable for our users, understanding pain points and opportunities.

Time

4 weeks

My role

  • Plan discovery work

  • Organize and facilitate workshops

  • Share learnings and progress

Who was involved

  • Stakeholders

  • Product Manager

  • Engineers

  • Customers

Internal problem evaluation

Before planning the discovery, we needed to understand what was done during the hackathon and what we knew.
To do so we adopted a framework to understand: 
What we know, What we think we know and What we don’t know.
This framework helped me raise awareness of assumptions we had, what we could quickly validate and support with data, and things we were not aware of.

Matrix.png

Research plan

As a result of the internal evaluation, it was learned we needed to plan discovery on two things:

  1. The already live feature Quality AI.

  2. Understand better the Developer Experience to identify and solve the right problem.

Thus I decided to approach research by dividing it into 2 stages with specific approaches.

  1. Get feedback on the Quality AI feature

CONTEXT

The goal of releasing this feature was to test users' reactions to this specific solution but also to validate if the suggestions provided by AI were valuable for developers, and lastly, how we could improve it. 

GOAL: Test Quality AI

Understand which feedback we have from early access users about Quality AI. Evaluate the effectiveness of Quality AI, that is usable and solves the problem. 

SUCCESS

3-5 interviews with active early-access users

METHODOLOGY: user interviews
Interviews.png

I chose this method because I wanted to gather information through hands-on experiences, ask follow-up questions, and better understand how our customers were perceiving the new feature.

This would help inform stakeholders with relevant information about the next steps.

Research
Findings

We had a total of 3 user interviews with customers that actively used Quality AI.

Interviews Findings.png

2. Identify pain points throughout the development process

CONTEXT

Since the Quality AI feature has been developed in a week, and it was mostly based on past customers' feedback and assumptions, there was a need to explore and understand the Developer Experience in the development process to answer two questions:

​

  1. Are we solving the right problem? 

  2. Did we develop the most valuable solution? 

GOAL

Identify the right problem throughout the Development workflow, if and how we can leverage AI to solve it.

SUCCESS

At least 10% of contacts fill in the survey

METHODOLOGY: survey
Survey.png

The main goal of this research segment was to gain insight into developers' actions, identify any patterns in the challenges they encounter, and obtain a reliable perspective on how users are integrating AI into their development processes or how they evaluate it to solve pain points.

CHALLENGE

The survey received a limited number of responses, and I required additional data to verify my assumptions. As a result, I conducted thorough desk research while the survey was still active, examining competitors' strategies and market studies.

Desk Resarch

How might AI fit into the Software Development Lifecycle?

The benefit of this approach is that the development cycle is unified throughout the industry. Therefore, the process that developers follow from planning to maintaining code has served as the foundation and guiding principle for my research.

I went through research done by industry top players and competitors' products.

Design
Findings

  1. Most of the AI tools are in the development and Testing stages of the workflow.

  2. AI performs better when it comes to completing repetitive tasks, also in code generation.

  3. It can learn from users' behaviour and replicate it

  4. 2 out of 3 users would find value in AI writing. documentation and even more in using AI to write Unit Tests.

  5. 2 out of 3 users would prefer AI fixes and suggestions in an IDE plugin

    This brings security concerns about how we treat data and where we send them.

  6. Better integration into the development workflow as suggestions in PR is too late.

Desk research.png

Ideation Workshop

To take action on the findings I organized a workshop with the entire team. The objective was to brainstorm solutions and develop actionable plans. I felt it was important to involve developers in this stage to assess feasibility and apply agile methodologies.

 

I made sure to keep the distinction between improving the Quality AI feature and developing a solution to solve pain points in the Development process.

Ideation workshop.png
Quality AI

  1. Group issues instead of creating a single notification per each issue

    Value:
    - faster ship (review, no pipeline wait)
    - reduce email “spam" 
    -     system response, less than 25 limit 
    - customer confirmation
     

  2. Notification settings allow users to decide how and what they want to receive.
    Configurable notifications. Specify categories and severities.
     

  3. Allow users to define which type of issues they want to receive suggestions.

    Value:
    - Custom solutions based on customer needs
    - Reduce noise
    - Faster commits to fix code

Development process

  1. IDE plugin to automatically create/update the unit tests (on simple things) while the developer is writing the code
     

  2. IDE plugin. Leverage codacy-cli and run the tools at compile time. Then make it possible to have suggestions directly while developing

The solution regarding a feature in the IDE/CLI was shared and handed over to another team working on a related project.

Concluson

Next steps

The next priority is to prioritize work based on the effort and value it delivers to customers.
At the same time, we should start a discovery process focused on building the first iteration of one of the proposed solutions.

bottom of page