Anatomy of a Prototyping Engagement: AnyCompany & AnyProto

Part 5: The Closeout

This post is part of a series walking through how I approach a PACE engagement end to end: the considerations, the decisions, and the consequences of getting them right or wrong. Details here are anonymized and composited for illustration; AnyCompany and AnyProto are stand-ins, not a real customer or product.

← Part 4: The Dev Sprints | Series index | Part 6: The Handoff and Knowledge Transfer →

Scene

After 6 weeks of work, lots of sync meetings, and at least 3 demos, AnyProto has reached a state that I believe achieves the success criteria. By this point I've gotten to know AnyCompany and the VP of Video pretty well.

After some final polish and a few demo dry runs, I declare a code freeze and turn my focus to the closeout deck, and everything I'll need to take care of afterward.

The closeout meeting has been on my calendar since the end of the scoping proposal meeting. I'm ready to really sell AnyProto as a solution and a stepping stone toward solving AnyCompany's business problem. The end of this prototyping engagement is near, and this next meeting is the capstone.

Now What?

I'll preface this section by saying I've probably been contributing to the closeout deck all throughout the engagement, as I converse with AnyCompany and develop the solution.

Certain sections, like the Demo, Completion Summary, and Path to Production, stay relevant throughout the engagement, so I write them as the information comes in. I do my future self a real favor by letting the closeout deck develop continuously as AnyProto progresses.

It's very easy to underestimate the work required for a prototype, and development tasks tend to compete with the closeout deck for my attention in the last week. I avoid crunch and quality degradation by never treating the final presentation as "the last thing to do."

Let's talk through some sections of the closeout deck.

The Goal, Objectives, and Success Criteria

These sections are largely the same as they were in the scoping proposal. I have to consider that the VP of Video may be inviting additional stakeholders to look in on his project. It wouldn't be a big surprise if C-suite leaders, or even the President of Video at AnyCompany, showed up to this meeting.

I have to plan my presentation with that in mind. It's important to keep these sections for posterity, but if new members from AnyCompany need an introduction to AnyProto, this is my chance to restate what the prototype aims to accomplish from a long-view perspective. I also make sure all the information is up to date. It's not impossible for success criteria to get amended, or even added, during development.

The Demo

After introducing AnyProto, I get right into presenting the completed version. I might be tempted to just demo the last bits of functionality the VP of Video hasn't seen yet, but I actually want to demo the full application. For one, there will likely be new stakeholders in the room, as I mentioned above.

But also, it's almost a certainty that the closeout presentation will be recorded and shared around AnyCompany afterward. The context of the entire engagement needs to come through in this meeting, in some form.

In the demo, I'm taking on the role of a salesman rather than a technologist. The technology is great, and I worked hard on it, but as a solutions developer, I'm unfortunately not in the business of showing off science projects. I need to communicate the value AnyProto adds for each persona it's built for.

"I authenticate using Amazon Cognito into a user group with permissions to access a subset of the video archives, then I send a query, which reaches Amazon OpenSearch Service via API Gateway and Lambda. The search pulls in S3 URIs and other metadata, which the web application uses to fetch videos and provide clips for me to peruse."

This is too technical, and it doesn't clearly tell the story of how AnyProto delivers an experience that serves AnyCompany's goal. Compare that to:

"I need to collect some b-roll footage for a docu-series about this event, so I use my AnyProto account to search for videos associated with it. I know I'll need footage of a few subjects of interest, so I search for that. AnyProto gives me a series of clips I can review, and when I find ones I can use, I add them to my cart for later purchase."

This version is better, since it's told from the perspective of an actual end user, and it points to the value add for both the user and the business. The user benefits because they've got a docu-series to create. AnyCompany benefits by helping that user and getting paid for it. AnyProto is the software that makes the transaction possible.

Another big tip for this demo, and demos in general: solicit questions and feedback periodically. I stay aware of how much I've been talking at any given moment, and pause when I notice I'm drifting into a tangent or about to go a layer deeper than I need to.

I need to get through the demo in a reasonable amount of time, but a real conversation about what I'm showing is welcome too. I just have to stay mindful of when to take control of the pace, so everything that needs to be communicated actually gets communicated.

Completion Summary

This is the section where I have to convince stakeholders that what we set out to do actually got done. As I covered in Part 3: The Scoping, a previous post in this series, choosing measurable, demonstrable success criteria (ones whose completion leaves little room for interpretation) pays off directly here. These slides tend to be data-driven: accuracy percentages, sample sizes, latency averages, eval breakdowns, and so on.

For AnyProto, a completion summary would include recall and precision scores, plus a report on the clip metadata returned from the sample set of videos and search examples. We're measuring segmentation error specifically against a 60-second threshold, so that needs to be stated explicitly.

AnyCompany will get the software soon enough to run their own testing, but it's useful for both of us if I make my tests replicable. That way, AnyCompany can't turn around after the engagement ends and say their numbers don't match what I reported.

In the same spirit, I need to be very clear about any assumptions or conditions built into my testing process. If AnyCompany later tests against a completely different sample set of videos and searches and gets different metrics, they need to recognize that test falls outside the scope of the engagement.

This may sound rigid, and there's a bad-faith version of this argument someone could make, but even so, I need to keep a collaborative spirit through this part of the engagement. First, I genuinely shouldn't be overfitting the solution to the sample data I was given in the first place. More importantly, AnyProto is just a prototype, and there's still work to get it to a production state, work I do have a supporting role in.

Path to Production

The Path to Production is a key part of the closeout meeting, since it impresses on AnyCompany that their time with AWS isn't over, and that we'll be there if they need guidance going forward. I treat it as "if I had 6 more weeks, here's what I'd do with them." It mostly serves as a product plan for integrating AnyProto into AnyCompany's existing infrastructure, and as a roadmap for future features.

All the architectural and feature possibilities I saw potential in during development go here. All the out-of-scope concerns and requests from the VP of Video and AnyCompany go here too.

Strategic suggestions that would benefit product teams and AnyCompany's account team belong here as well. If I was forced into any "quick fixes" or workarounds that don't quite follow architectural best practice, this is the section to flag those as needing attention.

The Path to Production shouldn't just be a laundry list. I put my solutions architect hat back on and get into prescriptive guidance, so AnyCompany is genuinely better equipped to tackle these tasks.

That looks like proposing architecture that builds on what's already there, and references to AWS documentation or blog posts about services or features that will matter for future development. A good Path to Production reads like a directory of tools a future contributor can use to take AnyProto further.