Call Center Village @ HTH 2025: Post-Mortem

This is an in-depth review of the things I think we got right, and a comprehensive breakdown of the things I know we got wrong.

Office Hours & Scripting Sessions

Before we get started, if you haven't seen our Office Hours and Scripting Sessions schedules for June 2025, you can check those out here:

Office Hours & Scripting Sessions Schedule - June 2025

We’d love to see you at one of our upcoming weekly meetings!

Call TheoryPatrick at Call Theory

Hackers Teaching Hackers?

Hackers Teaching Hackers (HTH), previously known as 614Con, was our first run of Call Center Village at a security-focused conference. We've done it at an industry conference, and to a private audience, but this was the first "general security" conference we've attended.

This conference was more technical than my general newsletter audience, so this article will likely be closer to a 1010 on the nerd-o-meter reading.

This year's HTH conference was a 3-day event at BrewDog DogTap in Canal Winchester, Ohio. As I understand it, the location does not change very often so you can reliably plan on it being there next year. I like local tech conferences.

It was our first time attending this event in general (let alone running our village at it.) We purchased all of the workshops and stayed at the DogHouse Hotel to make everything easier logistically and to try and, you know, learn something.

And learn, we did! While I'd love to dig deep into the workshops and sessions we attended, I need to focus this article on our experience with Call Center Village so it doesn't get out of hand.

Village Updates Since NAEO

NAEO 2025 was an industry conference where the "village" was setup in a completely separate building (but same complex) from the main conference.

I was determined to run Call Center Village at NAEO 2025 regardless of the format and audience of the conference. The goal was to get experience.

Good: because we got into more conferences!
Bad: we couldn't rely on direct foot-traffic during NAEO.

Regardless, the challenges we utilized at NAEO 2025 likely were too domain-specific (i.e., Amtelco environment-based) for re-use at HTH 2025. This means we had to update our challenges to better reflect a more technically capable audience - one that wouldn't have very much insight into the Line of Business (LOB) applications in question.

While updating the challenges, I also took steps to improve some of the functionality of our open-source challenge-dashboard we use at the conferences.

The changes are not yet published. We have a private branch with our changes since NAEO and the adjustments during HTH that we'll be cleaning up and merging with the main branch. This will be after we get Starbase published publicly, though.

Let's walk through the referenced changes.

The Village Countdown

I removed most of the extraneous information on the countdown page. This makes it look more ambiguous, and of course, more scary!!1

The first challenge is always to find the dashboard and unlock it. The countdown page is used in-case someone were to...find the challenge dashboard early.

Our village tickets contain two links. One of the links (also a QR code) goes to callcentervillage.com/hth2025. They have to decode the barcode on the front that says "Digital Entry Point" to find the second link, which is for the Challenge Dashboard instance specific to that conference.

Call Center Village - Call Center Village

2025 Hackers Teaching Hackers Conference

Call Center Village

Previously, the challenge countdown would show all kinds of information. Now it shows just a countdown. It worked out nicely displayed on a TV in the village.

Testing Improvements

As part of the annoyance of testing (i.e., changing the .env dates for the village to preview pre/during/post the event) I modified the views to be manipulatable based on environment and query string.

In testing environments, we used a query string called override set to past for when the village is in the past. We used future for when the village is in the future. Omitting the override query string will load the village as if it were during active hours.

The implementation is gross, but this isn't stable software (yet.)

Creating A Backstory

We added a backstory option to most challenges, which is a story-line that helps link the challenge to a real-world example of it within a call center. Essentially, it helps explain why the challenge is relevant to the village.

A Handful of New Options

We added a variety of different informational keys to the challenge JSON and incorporated them into the dashboard.

We have a new difficulty rating of easy|medium|hard and an array of tags (I stuck with 3 per challenge) that broadly explained what was being tested.

Finally we added a ctfd flag which indicated which of our challenges would reveal a CTFd flag for the larger Capture The Flag (CTF) happening at HTH.

Ticket Please

To better support the initial ticket challenge, we added functionality that allows the first person to claim the challenge to be marked as the winner. Once unlocked, the rest of the challenges will be available to work on through the dashboard interface.

This eliminates many "No, I was first!" arguments.

However, I already found a logic bug here by writing out this post: If two people find the form at the same time, they both will have the ability to submit the form (until the XSRF token expires.) This is one of those cases where the second-person submitting would win (by overwriting the first submission.)

I've got it on my list to fix.

The revelations above are a huge reason why I like to write detailed follow-ups. It helps me identify other problems that need adjusted that I might not have considered otherwise (until someone exploits them, I guess.) It's also the reason I don't use AI to write these articles.

Claim The IDOR Challenge, Too!

We updated the Insecure Direct Object Reference (IDOR) challenge so that the same mechanism can be used to claim the win as is used on the Ticket Please challenge.

I expect this functionality (the ability to claim the challenge win in some form) to be the default for all challenges eventually.

A Filesystem Approach

The challenges have two different states that can be managed to make the dashboard interactive without adding in databases. Essentially, the configuration file is loaded using the standard Laravel way (config/challenges.php) and then merged with results loaded from the storage path.

This merged result only exists in-memory during the request, but can be re-built using the config file and stored results.

This gives us a simple way to track state. We currently track winner and status of the challenges, and (fun fact!) this is the mechanism I identified needed fixed from the Ticket Please section of this article.

I also identified that I'm going to encrypt/decrypt the content of those files in the next version. This ensures someone can't manually manipulate the files without at least knowing the application's encryption key.

I want to keep things config-file based (and not in databases or in-memory cache) until I absolutely have to change it. In theory, I am reducing the infrastructure's attack surface by not including database services.

However, I am increasing the application-logic attack surface. Expect this to be better handled before we approach a 1.0 release.

Clue: The Village

Another feature addition that we made to help tie-in all of the story-lines created across the challenges was to build a clues key into the challenge JSON. The key of each object corresponds to the key of another challenge, and the value of each object will be the clue you want to reveal.

This was synced up with the results from storage and merged during each request. The views were then updated to display clues if the particular challenge had been completed.

This effectively allowed me to reinforce a cohesive story-line and reduce the barrier to harder challenges as the other, easier challenges were solved.

The Admin API Is Born

As I was getting ready for HTH 2025 and I published the dashboard, I realized that in order for me to update the challenges in the same way I was using (via the local filesystem) I would have to SSH in or otherwise remotely access the console of the server.

This was technically possible, but wasn't ideal as I wasn't ready to rely on cellular or hotel WiFi for a place I've never visited. Plus opening SSH to a con full of hackers felt like a bad life choice and I didn't have a jump-box ready to go.

So, I made a simple administrative HTTP API that uses an API token defined in the environment file to allow updating the status of a challenge, or deleting the status associated with a challenge (i.e., resetting the challenge back to the baseline definition in config/challenges.php)

This allows us to update/fix any challenges throughout the course of the village using just an HTTP request.

These changes were made incrementally over the last several months, but I spent more time than I wanted over the 4-weeks leading up to HTH testing everything.

Imposter syndrome strikes again, I suppose.

Call Center Village Begins!

We were able to setup the village the night before, which was much appreciated. I didn't want to have to get up extra-early – the 7:00am wake-up was already too early for me.

Here are a few photos from the setup.

Throughout the day, I found myself giving the same "introductory" speech over and over again. I'm thinking perhaps (in the future) I put that behind a phone-number (i.e., a recorded message) so attendees can get a more consistent welcome message, also allowing Jay and I to focus more on village operations.

Not saying I wouldn't give an introduction still to anyone who asked, but for those who might be a little more shy (or if we get stuck helping others) they can get an idea if it's something they would enjoy without any anxiety.

Dashboard Updates

We were able to make updates to the challenges throughout the day, so basically everything went to plan in that regard. I loaded up Postman, updated the API key for the live dashboard server, and was able to keep things in sync (since many challenges required attendees to show me that they solved it.)

Perhaps except for my spelling/typing skills, which seemed to deteriorate throughout the day. The consequences of the con being at a brewery, I suppose.

Sir Fails-A-Lot and his Village of Fail

Before I start working through a list of all the things that went wrong, I want to reiterate in-case it wasn't clear: Jay and I consider this an immensely successful village run despite some of the failures we'll describe below.

It was night and day better than our NAEO run due to the audience's technical skills and our village's proximity to the con. Plus the CTFd flag.

A Signed Failure

My plan was to print out signage as photos and attach them to the cheap foam-board you can find anywhere. The photos came out really nice, but none of the 5 different stores I checked had foam-board other than tri-fold.

Since I left this to roughly the last minute, I had to improvise by using the painter's tape I brought to keep the tri-fold extended so the picture could be attached as a sign.

Also, I may or may not have dropped the photo and crinkled it a little bit. Twice.

It looked fine from a distance, but lacked professionalism the closer you got. An easy enough fix for next time.

I also had a 3 smaller signs that denoted the Data Center, Call Center, and Remote Agent areas. These too had the same foam-board issue (meaning I could not find any.)

Instead, I bought a twelve-pack of poster paper and layered 4 on each area's sign.

It was...fine.

But the best thing moving forward is to get these printed directly on foam-board. This is fairly easy, it just requires me to not wait until the last moment to take care of it.

Absolute Timer Confusion

We scheduled the event time for the village to start at 9am and go until 5pm. So when the countdown reached 9am, and the conference organizers started going through the welcome messages and orientation for attendees, the dashboard started counting up.

But there was no indication of why it was counting up, so it didn't make sense at first glance.

Jay pointed out that it would be an easy way to illustrate how long it takes for the community to unlock the challenge dashboard from the initial ticket challenge. I agree with him, so I will likely make that have a different color or other visual indicator to explain why its counting up.

The Ticket Tax

At all of our other events, we've been able to distribute the tickets (that kick off the first challenge) to every attendee fairly easily because we sponsored those events. We would just put the ticket in the bag that every attendee gets!

We didn't have that opportunity at HTH 2025 (we went as an educational village, not as a commercial sponsor) and I didn't want to ask them for that favor.

So essentially, we set out stacks of the tickets at every table (within our village) and explained that they needed to start there. It mostly worked, but I brainstormed a few options to better handle it in the future:

1. Set the ticket stacks on the unofficial "sticker/info" table that people made at the entrance of the con, catching the attention of attendees as they entered.
2. Jay or I could promote our village and hand them out to people as they walk-in as another way to get more attention to the initial ticket challenge.
3. Use the extensive networking opportunities available before and during the con to distribute tickets manually.
4. It was a waste of a lot of paper (we printed too many)

Doing this at DEF CON 33 would be expensive (depending on how many tickets I would need to print) so I will likely change the ticket challenge (or get rid of it) when we do the contest there.

The Mobile Display Cutoff Failure

I did a lot of work to ensure that the dashboard worked well displayed in full-screen mode on a typical monitor and television. I took less-care on the mobile version (insert my excuse here) assuming all the hackers would be using laptops.

Narrator: They weren't.

While almost everyone had a laptop, the barcode scanning was being done on the mobile phone. As a result, when they "figured it out", they were directed to the dashboard page on their phone. It worked...for like the first row of challenges. After that, everything was cut-off because of overflow: hidden CSS rules.

I didn't have time to debug this live at the conference, so I essentially told them to use a laptop (one of the villages, or their own) to access the site. We'll get this fixed for next time.

The Dashboard To Device Pipeline

Once the dashboard was initially unlocked, the full dashboard would display on our TV in the village. However, it was difficult to "move" people from the visual dashboard to being able to load it on their phone/laptop to explore the different challenges.

Perhaps a QR code on the dashboard display to easily get it on your device?

This will only get harder as the village subdomains will likely be more obtuse as we go.

The Payload Problem

As part of the USB Rubber Ducky challenge (lovingly called, "Rubber Ducky, You're The One") you had to activate a USB rubber ducky and load a payload from the Hak5 community edition of PayloadStudio or from their pre-built payload list at payloadhub.com.

We fought with intermediate networking providers for a while who seemed to be blocking the "malicious" payloads. We had this problem on the official Hak5 domains and trying to use Github to obtain the inject.bin in question.

A village attendee cleverly figured out a workaround which I'm going to try and incorporate into this challenge for the future so we don't have to worry about network conditions. (I'll let you know what that is when I figure it out myself.)

Conveniently, I didn't have these problems testing from my own environment.

The CTFd Flag of Fail

So, I've never really participated in a community Capture The Flag (CTF.) I've done a lot of simulations through TryHackMe, HackTheBox, and others.

So when I was approached by HTH to have some sort of tie-in with the CTF, I was excited! I figured I could put the flag behind our hardest challenge: The Call Center Job. Whoever gets the flag first wins!

Wait, whoever gets the flag first wins? That's not right, apparently.

The reality is that every player/team would theoretically need the opportunity to participate in the challenge and obtain the flag.

Okay...but that means we'd have every team trying to call the call center for a full-follow through, putting the call center agents through a more intense regime that I had intended and explained to my client.

As a result, I moved the CTFd flag to be part of the "IP Phone Defaults" challenge, which involved finding the phone on the network and logging into its admin interface using the default credentials. The flag (in the format HTH{xxxxxxxxxxxx} was entered as the name of the SIP account on the phone, so they had to navigate through the web interface and find it.

This worked well, was easily reproducible for every team, and wouldn't cause issues with the (real, actual, live) call center. I'm going to have to go back and re-design the "The Call Center Job" challenge to work for many, many flag attempts.

Remember, my expectation is that Call Center Village is a "community" effort and that everyone is working together. The reality in the broader CTF is that teams/players are working relatively independently and in most cases against others. Can't we all just get along?

Factory Reset For Factory Credentials

We brought two SIP phones. One was meant to be logged into via the admin UI via the default credentials, and the other was supposed to be attached to our Asterisk server for use with the SIPVicious challenge. Apparently, I didn't bring a Display Port to HDMI adapter and the expected automatic DHCP wasn't working so I couldn't get our Asterisk box online.

Also, I didn't test it the night before where I might have had enough time to fix it.

Instead, Jay had a good idea to explain it away as a decoy phone. Most people ended up using the phone's physical interface to find the IP address because they had never worked with an IP phone before, but they were also easily scanned via nmap from our networked laptops.

Anyhow, someone didn't like that the default credentials weren't working on "the decoy" so they factory reset the phone so that the default credentials would work.

However, that would remove any flag programmed (which we hadn't done anyways) so it basically just made it a harder decoy. They would search and search and never find the flag, because it only existed on the other phone we programmed it into.

We gave a few folks some additional hints because of this unintended frustration, but I think I do like that for more difficult versions of this challenge.

Remarkably Bad

We had a challenge that involved finding a PIN code for the answering service documented in a note on a Remarkable2 device (that my wife just gave me as a hand-me-down!)

This challenge was linked to the IDOR challenge, which when found, revealed the actual PIN code to access the Remarkable2 tablet.

Crickets...nobody even noticed it was a thing.

Towards the end of the day, I effectively shoved it into someone's face and said "here's this challenge, here's how it works, boom: you win the medal!"

The response was, and I quote: "What was the point of that?"

And they were right. What exactly was the point of that? I had envisioned a much more cohesive community effort that would have pieced these things together, but overall I'm going to drop this challenge because it as a pretty big flop. I can always revisit later if we find a more compelling way to integrate it.

For now, I'll enjoy my cool Remarkable2 tablet. Fun fact: If you are logged in and someone else uses it, they can get the root SSH password from the settings page. If its on WiFi, they can then remotely access your Remarkable2. It requires some device knowledge and social engineering (or lazy opsec) but it is something to be aware of if you store anything of importance on it. (Also make sure you enable device encryption.)

We Are Currently Clean On OPSEC

Our OSINT challenge wasn't great. It effectively made them find the AnswerTarget website, and read the FAQ. The tie-in to the overall challenge was explained away in the backstory, but wasn't otherwise cohesive.

It was also linked explicitly with the eventually cancelled "The Call Center Job" challenge so there wasn't a lot of incentive to spend time on it.

In any case, it was way too easy. I visited the OSINT village for a few minutes and got some ideas, so we'll be either improving this or removing this one in the future.

A Garden Variety of Failures

This is a less detailed list of other issues I identified that I think we should improve moving forward:

• The Sparrow's Snap Gun challenge (a kinetic attack against locks of all kinds) was extremely hard. It needs a difficulty level increase from medium to hard.
• I brought my Samsonite thrift-store briefcase which a lot of people enjoyed unlocking. It wasn't explicitly part of a challenge, but I put one of the orange keys from the Money Bag challenge inside. (Though, only one person put that together.)
• The clear-text log files did not generate correctly and I did not double-check it before the village started. Essentially, this flag didn't exist. I watched a hacker bravely fighting the log files (and even finding credentials - just not the ones I put there) so I gave him the win and will have a test built for this.
• The Decompile .NET challenge didn't include a flag. Hackers like finding flags. Without a flag, why would they do it? I think this means I'm going to have to create my own .NET application following this bad practice where I can compile in a hard-coded CTFd flag.
• Individual challenges could have been marked (in physical space) better. I don't think it was very clear where to go after identifying a challenge you want to try on the dashboard to the location of it in the village. We did our best to guide people manually.
• We had our butt-set, but didn't have a working copper phone system to showcase how it works. I'm going to try to find some 66-blocks somewhere and build out a physical setup for it.

Our Other CTFd Issue

Our CTFd challenge was marked as hidden on the CTFd portal, so hackers who completed our CTFd flag challenge (the "IP Phone Defaults") would be rewarded with a flag they didn't know where to turn in.

There are penalties for incorrectly submitting flags, so it came up quickly.

We had originally submitted the name as "The Call Center Job" but then pivoted as explained earlier. This means our new challenge "IP Phone Defaults" wasn't what they should be looking for.

Shortly thereafter, additional folks asked about it and showed me that they couldn't find it on the actual CTFd flag page. We even did a ctrl+f page search for various terms related to both challenges.

So, off to CTF Support who helped identify that the CTFd platform had it marked as hidden. A quick button click or two and things were back on track and teams could turn in our village's flag!

Hackers Love Flags

This is something that I really, really didn't understand fully. While the hackers were participating in the larger HTH CTF, Call Center Village is essentially a small-scale CTF itself.

However, I didn't build in any actual CCV{xxxxxxxxxxxxx} type flags into Call Center Village (CCV.) It was more of a "verify with the onsite coordinator" type system. A lot of hackers did not like this, and felt like the incentive to "learn the process" wasn't rewarding without being able to claim a flag of any kind.

So I promise: next time, there will be flags. Plenty of flags. FLAGS EVERYWHERE, OKAY?

A conference we hope to return to

We had a really good time, learned a lot, and met a lot of new people. I got the impression that we should try and submit our village again next year, so that's the plan. (We didn't bomb so bad as to ruin our reputations.) We also got tons of feedback to help prepare us for our DEF CON 33 run as a contest.

Remember, you should stop by our DEF CON 33 contest and come say hi at Party-Line, our party at DEF CON33!

We also met someone from Queen City Conference in Cincinnati and were encouraged to reach out and submit a CFV. Hopefully we'll know more about that later this summer.

Queen City Conference - Home

Queen City Conference is a 3 day cyber security conference that is held in Cincinnati Ohio between November 7th - November 9th

For now...

We're back to focusing development efforts on Starbase. Public release is almost here, I promise. No more pesky security conferences to distract me.

If you read this far along, thank you - "You the real MVP."

Final Results

Multiple challenges were marked off due to technical issues, including "The Call Center Job" and "Asterisk to Ashes."