Anatomy of a Prototyping Engagement: AnyCompany & AnyProto

Part 3: The Scoping

This post is part of a series walking through how I approach a PACE engagement end to end: the considerations, the decisions, and the consequences of getting them right or wrong. Details here are anonymized and composited for illustration; AnyCompany and AnyProto are stand-ins, not a real customer or product.

← Part 2: The Discovery | Series index | Part 4: The Dev Sprints →

Scene

Discovery is now done, and I have about a week or so to turn around a scoping proposal slide deck. The scoping proposal is very similar to a product requirements document.

Its role in this engagement is to get me and AnyCompany aligned on what's going to be built, what isn't, what the goals and objectives are, what success looks like, and what's on the table to pursue them.

I'd argue this is the most important step in the entire engagement, since it sets the tone and pace for everything after it. Scoping is where a good solutions developer sets themselves and AnyCompany up for success, and where a novice might hang themselves out to dry for the next month and a half (speaking for a friend, of course).

Fortunately, I have quite a few things supporting me:

Supporting documentation regarding AnyProto from the account team
A recording of the discovery session
Notes from the discovery session, clearly defining the key details and a direction for research
Opinions and guidance from my team and the account team
My own AWS expertise and experience to back me up

I have a few days to really sink my teeth into this problem and use case.

Now What?

It's time to do plenty of research into existing solutions to similar problems and relevant technologies. I'll start there, then draft some solution architectures to review with peers and seniors. It's also worthwhile to have short huddles with the account team to validate my potential approaches.

Let's go over the different parts of the scoping proposal and how I tackle each.

The Goal and Objectives

I need to restate what I've discerned about AnyCompany's long-term vision, and the nearer-term roadmap to get there. Get this right, and AnyCompany walks away thinking "okay, Joseph's been listening to what we're after, and he's going to help us get there." It's easy to lose trust here, since by this point AnyCompany has spent a lot more time and resources getting to this point than I have.

Fortunately, if I've been listening even a little to the VP of Video, I'll know the ultimate goal this prototype is a stepping stone toward: creating new streams of revenue for AnyCompany by monetizing the vast archive of data they currently have stored, via a clip marketplace for consumers.

I think of the objectives as the milestones for this specific prototype, all in service of that goal. For AnyProto, I know I'll need to at least:

Intelligently enrich video archives with metadata, so users can perform natural language search to find their favorite videos.
Segment video archives by scene, to accurately create clips.
Create a user experience and API layer for users to access and view clips and their metadata.

Already this seems like a tall order, but I have to remember this is a prototype. I have to prove the functionality, not necessarily build a production-ready application. There's room to scope down, and I will.

Success Criteria

This is usually the trickiest part of the scoping proposal, and also the most important. Success criteria are how, in the end, I'll prove to AnyCompany that my solution works. It's important that the criteria be relevant to the objectives, and something both the VP of Video and I agree on.

Ideally the criteria are measurable, or at least not subjective. I'm trying to strike a balance between attainable and provable. It's fine to have some "checkbox" style feature criteria, but the less wiggle room for scope creep, the better.

The worst version would just restate the objective, something like "segmented video archives by scene to accurately create clips." That says nothing about how accurate we need to be, or what accuracy even looks like.

A better version reads: "dynamically segmented clips from a sample of 20 videos match the control set with at most 60 seconds of error." That's a lot more scientific while sticking closely to the objective, and I already have an idea of how I can measure it. Good success criteria stay relevant throughout the whole engagement.

Some of the success criteria I used for AnyProto:

Dynamically segmented clips from a sample of 20 videos match a control set with at most 60 seconds of error
Natural language clip search provides results with an average precision score of at least 90% across a sample set of queries
Natural language clip search provides results with an average recall score of at least 80% across a sample set of queries
The web application presents, for each clip:
- Clip video playback
- Transcription
- Subject tags
- Related videos

Most of these are measurable, or at least indisputable in their conclusion. The difficulty with some of these criteria is the actual numbers. It's intuitive to measure precision and recall for a search use case, but how accurate is acceptable? How inaccurate is unacceptable? Ultimately, it would be a risk for me to decide that on my own.

Again, the point of the scoping proposal is to come into alignment with AnyCompany, and success criteria are a great place for a back-and-forth negotiation on how high to aim.

The VP of Video likely isn't familiar with precision and recall calculations, so surprising him at the scoping proposal with scientific questions will probably get a "let me think about it" response, not especially helpful.

My guidance: work with the account team to come up with initial estimates, and present them to AnyCompany as an opening bid they can adjust before the meeting ends.

User Stories

Time to break out some UX/UI design skills. For the scoping proposal, I'll want to present UI mockups alongside written user stories. The visuals don't need to be high fidelity, wireframes will do, and are often better, since they leave more room for the customer to imagine and fill in themselves.

Each user story should correspond to an objective, and sometimes a sprint, so they'll loosely tie back to a success criterion as well. User stories tell the story (when, how, and why) of a defined persona interacting with the web application, and how that application responds to their inputs. Here's an example set:

As a user, I log in to AnyProto using my credentials and use the main page to search for clips involving a hometown parade from around 1997, so I can collect footage for my "this day in history" segment.
After the user searches "hometown parade 1997," the query is used in OpenSearch's search API to find the top 50 closest matches with a minimum score of .65. The API Gateway endpoint returns surface-level metadata, database primary keys, and Amazon S3 URLs for video thumbnails.
As a user, I'm presented with at most 20 thumbnails for videos. I use sorting, faceting, and filtering to find the videos of most interest to me. After finding one whose title matches what I'm looking for, I click on it to inspect further.
An API call is made using the video's primary key to fetch further data: the full clip URL, transcription data, and tags. The retrieved data is presented back to the user in an interactive video modal.

Alongside each written user story, I add a low-fidelity mockup or wireframe of what the main page, search results, and modal might look like. There's no commitment on my side yet, but these examples will likely stir AnyCompany's thinking about how the web application should look.

Proposed Architecture

Now it's time to put my architectural skills into practice. This is explicitly a first draft (nothing's actually built yet), but it's still important to estimate it as well as I can, because this is the architecture I'll begin building during the build phase. It's also a good moment to learn early if I'm planning to build with a service that's on AnyCompany's restricted list.

When presenting the architecture, I make sure to loop each component back to the user stories and objectives, so non-technical viewers can connect the dots and understand my choices.

I'm also ready to defend those choices if needed. I've already considered alternative solutions I'd be open to, and I come in with opinionated guidance. AnyCompany came to me for my expertise and judgment, after all, so I should be assertive when I believe I'm making the right call.

There are plenty of ways to clip videos into segments and index them: here's one potential architecture for this use case.

Proposed AWS architecture for semantic video search.

Bill of Materials

The most difficult and consequential parts of the scoping proposal are behind me now, but the rest still matters. The biggest consideration here is the resources required from AnyCompany to get development started. For AnyProto, the obvious piece is a sufficient sample set of videos, but also a sample set of example requests AnyCompany's end users might send in.

This part is genuinely important. I need to get stakeholders on the hook as quickly as possible for delivering the materials I need. Delays here really slow down the actual build.

Imagine I have to proceed without their videos. I'd have to pull content from other sources (maybe download samples from the internet), but that content could be meaningfully different from AnyCompany's actual videos. The videos I download myself might have clear scene delineations where the real sample videos don't. I might approach the development differently as a result, ultimately wasting time and effort.

Assumptions and Risks

As I architect and write user stories and everything else, I inevitably come across assumptions: either implicit from the context of the engagement (the media content being ingested will be entirely video files; audio-only or text-only isn't supported) or made to keep my development work moving faster (we're only supporting English for now).

These boundaries are important to call out explicitly, since the VP of Video may be making different implicit assumptions about the project's scope. A big lesson I've learned prototyping for enterprise customers: assumptions are ruinous.

Say I didn't call out that AnyProto only supports English. The last-week demo rolls around, and the VP of Video says, "I have this set of international videos too. These need to be supported for me to get AnyProto off the ground." That's a whole sprint of delay, right there.

Or imagine getting that message after I've already delivered the prototype. "Hey, where's this feature?" (for something I never explicitly discussed or demoed, but assumed was included) is never something I want landing in my inbox. Surfacing these assumptions early greatly de-risks the engagement, and documenting them is useful for posterity too.

The same logic applies to risks. I want a collaborative approach to addressing anything that could jeopardize the engagement: a tight deadline that suggests additional resources are needed, or a difficult spread of time zones across stakeholders and contributors that makes it hard to get everyone in a room at once.

Proposed Schedule

This is where I lay out the sequence of tasks for each sprint on a timeline. It's largely estimation, and I've never had complaints about adjusting priorities mid-engagement. The parts that matter most, though, are the dates for demos and syncs.

Keeping the VP of Video engaged means meeting with him for at least 30 minutes every week: highlights, lowlights, learnings, or scope changes. Keeping stakeholders in the loop keeps them engaged with the project, and tends to lead to better outcomes. Depending on personality, the VP of Video might end up contributing real feedback that improves the quality of AnyProto.

It's also important to emphasize the end dates: when I'll stop building, and when AnyCompany can expect to receive the code. Leadership at AnyCompany will need those dates to plan accordingly.