Post-mortem: Call Center Village Contest @ DEFCON 33

Despite heat that we kept referring to as "oppressive" along with wildfire-caused air-quality issues making just existing that much more difficult, we somehow survived 10-days in the desert surrounded by a bunch of hackers.

And hopefully we'll get to do it again next year!

This will be a long read that includes a variety of topics related to the Call Center Village contest infrastructure and logistics, including all of the things that went right at DEFCON 33.

Of course, we'll also breakdown what went wrong (and a-lot went wrong) including our Day 2 pivot. Here's what you can expect:

• Contest Setup
• The Day 1 Disaster
• The Overnight Pivot
• Moving the village forward
• Live Operators or AI Agents?
• Call Center Village Party ("Party Line")
• Final Thoughts

I've decided to skip going into details about BSides Las Vegas and SkyTalks. If you're interested in anything about those conferences with respect to our experience, please reach out directly. [FWIW: we had a great time.]

Contest Setup

Normally when you purchase a ticket/badge for DEFCON, you get in line. It's a whole thing called linecon and its recommended to pre-purchase your badge or get there early if you want to avoid the "paper" badge that you get when they run out of real badges.

This happened as of 8:30am on the morning of registration this year, partially due to logistics (same issue every year I've attended...)

However, we got to mostly avoid the pitfalls of linecon because inhumans get their own pickup spots!

Inhumans are those that are attending as creators, musicians, goons, villages, contests, exhibitors, sponsors, speakers, etc..

The real challenge for us was figuring out where everything was to get picked up. Since we were putting on a contest, we had to pickup 4 of our badges at the Contest Stage. Since we are putting on a party, we had to pickup 2 of our badges in the Ops Room up on L2 w207. Finally, we had to make our way to the contest location with all of our stuff in W1 210.

We entered from the North Lobby, which was a considerable walk in itself.

These are some of the farthest possible locations from each-other. I had Jay and Brendan hang out with our massive pile of rolling suitcases while I ran laps around the building trying to get all of our affairs in order.

We were already exhausted from pushing/carrying 300lbs of gear from the hotel to the convention center in 110 degree weather. That paid move-in option I declined to save money was looking mighty nice at this point.

After getting all of the badges needed for our party and contest area, as well as the prepaid badges I had purchased for family, we headed over to the contest area to find our W1 (Hall 1) Booth 210!

And there it was: Our phone booth!

We ordered and shipped a real phone booth and it was waiting for us in the shipping crate! We just had to unbox it and lift it vertical!

One thing we learned the hard way was the table linens. We had instructions to pickup linens (basic black linens) for the primary tables in our village/contest area. Not every table needed linens, but to make things look nice we wanted the front ones where we had stickers/giveaways/information to look professional.

When we requested the linens, they gave us DEFCON branded linens. I didn't object because I thought it was cool (and more importantly, I didn't know it was wrong.)

However, there was some questioning from contest staff who eventually let it go and said we'll deal with it if needed.

Narrator: It was needed. We'll come back to this in a bit.

One thing we did not have ready to go was "power." Like, we had power - we could see the plugs. But we were told we had to wait for the electrician to come by and connect things. As a result, we couldn't do a whole lot other than organize.

We ended up heading to lunch waiting on power.

After we came back, we continued with everything we could without power but eventually called things around 6:00pm. Because we had things like Toxic BBQ and Lucy Darling to get to!

That night around 8:30pm we got a message saying the electrician was going around so we made the determination we'd get in early at 8am (2 hours before doors) and connect everything that needed power.

This was absolutely cutting it close by any meaning of the phrase, but I didn't have an option other than dragging everything back to the hotel to do the "online" setup and then dragging it back again. Did I mention the 110+ degree weather?

The Day 1 Disaster

I got up early and drudged the 100 degree weather that existed at 7am for some reason and made it our contest space by 7:30am.

The power was connected. It was on.

While we were physically setup for the most part, we had to get the local network going, connect our servers and laptops, enable WiFi, setup IP Phones, configure ATAs (Analog Telephone Adapters), and setup the phone booth for our eventual Live Operator challenge. We also had to connect our WiFi to the venue WiFi since we did not get a network drop.

After getting all the power connected, we began to configure equipment. However, a problem we talked about earlier would come back to bite us: the branded linens.

The Contest Lead came by and let us know that the DEFCON branded linens were meant for stages and official DEFCON stations. This means we had to remove the branded linens and replace them with regular ones.

This meant we had to remove everything we setup so we could take off the linens. Pretty frustrating - and I think Jay took it personally - but we managed to only get setback about 20-30 minutes for this one.

This is where things really started to go wrong.

Now that things were finally setup and connected with power and WiFi (and of course the correct linens), we were able to see the full scope of problems ahead of us.

For observability, we had two OpenWRT devices connected by a Firewalla Purple that gave us network insights into activity on our local, private network. It also gave us some level of control if we needed to close out network connectivity to the devices but not ourselves.

The first of the two shipped OpenWRT devices was physically damaged, and the antenna had been removed/ripped out. I was hoping it was just the antenna (which we don't need broad coverage for our small challenge) but it ended up being the entire device. Now we had to use the second OpenWRT device as a WiFi repeater and we couldn't connect the Firewalla Purple as a router, meaning we'd have to setup the network from scratch (including DHCP.)

While not hard, the contest area opening time was coming up soon but I already knew a delay was in store. So I'm just a little freaking out with everything we needed to do.

This also meant we couldn't leverage the previously configured WiFi networks and had to reset the network to build it again from scratch. I ended up dumping one of the "easy" challenges that was to connect to a WiFi network that used vendor defaults as a result of this failure.

We also ran into issues with the challenge dashboard where whoever solved a challenge would "claim" the first solve flag. And another mobile display issue. All things we had to quickly fix (and things I swear I had already fixed.)

Next, our IP Phone wasn't working. It was actually hilarious, because we had the following people try and fix it with absolutely no luck:

• Me - A person who has used and is familiar with the phone
• Jay - A person who knows more about Asterisk/phones than ChatGPT
• Redacted - A semi-retired (in their 30's) phone service provider
• Redacted - A credentialed Electrical Engineer

A few others even jumped in to give it a go with no luck. The device itself, after being stripped down to the circuit boards trying to factory reset it, was donated to the Hardware Hacking Village for further disassembly.

How many hackers does it take to fix an IP Phone? We may never know.

The main problem with losing the IP Phone challenge was that it effectively gated the contest. The flag unlocked a clue required to solve additional challenges, and I did not plan or expect to have to deal with multiple paths due to failures. A big, dumb, hard lesson to learn in the middle of DEFCON.

Finally, my mini-PC that was supposed to be our Asterisk/Application server for attacking was physically dented and lost many of the back-end connectors. It was DOA. This was pretty bad too, as now our challenges were down to Physical Security and Social Engineering.

I worked on repurposing a laptop to take over this role, but got interrupted enough (by wonderful conversations from visitors) that I gave up on this effort.

I spent the rest of the day on Friday trying to cobble together a set of working challenges by shifting the flags/clues to different challenges, removing challenges based on broken equipment, and fixing the dashboard to accommodate all of these changes.

The effort was dull at best, and we started informing people to come back tomorrow, giving us enough time to address the failures.

Jay and Brendan kept visitors entertained (and learning) by walking them through all the equipment we brought, and the phone booth was a great conversation piece. Jay eventually got the phone-booth's phone to actually work (off his laptop) and a fun set of local extensions working for people to have fun with.

Despite the non-functional contest, we were able to provide some value to those who stopped by.

The Overnight Pivot

We left conference on Friday at 6:00pm, which is when Humans are required to leave. Contest staff can stay behind till 6:30pm. The DEFCON contest lead saw us leaving and was like "Hey, your humans have to leave, but you don't!" but I was too embarrassed to explain our situation and was hell-bent on figuring out how to make the contest work for Saturday, arguably the biggest day of the conference.

At this point in the day, it was mostly too late to find the equipment locally or even guarantee an overnight delivery in time. I was at a loss of what to do, and I spent most of the night thinking about what options I had available.

I finally landed on a pivot: leaning into the AI agent challenge.

git checkout -b major-trucking-pivot

If you recall from the last Call Center Village update, we added a second "capstone" challenge which was to socially engineer an AI call center agent. This worked in tandem with our live operator capstone to do the same thing.

If you also recall, I mentioned when people saw Call Center Village contest, they were expecting something closer to a Social Engineering challenge rather than a Blue Team contest for small business call centers.

As a result, I decided to go all in on that route, creating a number of new challenges based around specific roles within a company. Each of these new AI agents had their own voice, personality, hints, unlocks, flags, and knowledge-base. Together, we were able to create a (mostly) cohesive story-line where you had to socially engineer different AI agents to unlock information to help with the live operator capstone.

Some examples of the agents we created:

• Security Operations Analyst - Quinn
• Data Center Operator - Riley
• Facilities Manager - Sam*
• HR Assistant - Casey
• IT Help Desk - Jordan*
• Front Desk Receptionist - Taylor
• Training Coordinator - Pat*
• Sales Account Manager - Morgan
• Field Service Technician - Jamie
• Remote Support Specialist - Alex

*Unfortunately, we had 3 of the AI agents get flagged and blocked. I was hoping ElevenlLabs would quickly reclassify them, but they didn't get back to my appeal until Monday after the conference was over.

The new challenges worked surprisingly well. They were fun, challenging, but approachable. I believe a big reason for that is that when you call a real person, they have memory - but when you call an AI agent, if you screw up you can just call back and start over.

It's like the social engineering village but for those with social anxiety.

However, we ran into another issue as a result of our pivot: bandwidth.

Running off of tethered cellular service was...rough. I wasn't able to figure out an easy way to get my OpenWRT router onto the protected DEFCON WiFi network (it required installing custom certificates and software that didn't come standard on OpenWRT) and going on the open DEFCON WiFi is a gamble. The cellular service worked for a while, but it certainly felt like we were competing with the other 30,000 attendees for service.

We had loaded the ElevenLabs AI Agent Chat Widget into each of the relavent challenges, so you could chat or call via the computer to work on the challenge. However, we found that the connectivity was so spotty we were getting errors.

Luckily, we know a few things about phones and hooked up the AI agents to a live telephone number, allowing users to call from their cell phone where they generally did not experience bandwidth issues.

To be fair, you don't need to know anything about phones to make this happen.

I did crunch through enough LLM credits that I had to upgrade my plan halfway through the day. It turned out to be well worth it, although I think the billing interface that ElevenLabs provides is too difficult to figure out what a call actually costs.

We ended up getting about 16 teams try out the new format and got a lot of positive feedback - so much so that we're going to continue down this route.

Moving Call Center Village Forward

Besides the pivot from general security to Social Engineering, there were also a number of changes we are going to implement based on feedback we got on both Day 1 and Day 2.

For example, the ticket challenge was underwhelming. Without a physical ticket and just a sign that says to start on the website, people got confused. Partially because it wasn't clear where the Media Server was for DEFCON attendees to access. [Note: This was my fault.]

I also found that many people gave up if they tried to use their phone to scan the barcode and it didn't work. (It wasn't a QR code, so part of the challenge was figuring out how to scan it like with an app.)

My solution to the ticket-please challenge with the new format will be that you get started by calling a number and getting the website address from a simple/easy/basic AI agent through social engineering.

Diving back into the AI agent side of things, I plan on expanding those AI bots to be much more dynamic instead of relying on a static knowledge-base and system prompt. We're also not abandoning live operators in any way, as the capstone challenge will remain to socially engineer a live call center operator.

Live Operators or AI Agents?

The AI agents were a lot of fun, but almost everyone was able to break them and retrieve information they weren't supposed to give out. While this is partly intended, and partly a setup/skill issue that we'll improve on, we didn't have the same success against the live operator.

While the number of attempts were low, the live operator agents never gave out information they shouldn't have.

This was a huge success for my clients who participated and I want to thank all of you so much.

This also led to a situation I didn't expect: a four-way tie.

To resolve the tie, we awarded the win to the handle/team who first solved all 10 challenges. Congratulations, excenter!

Call Center Village Party ("Party Line")

While we didn't get everything we wanted (the phone booth moved to the party room, or enough power to connect all of our phones and servers) but we had a full room most of the night!

We had neon phones of different colors, tons of stickers, and even did a Black Badge Trading Card Game card swap (thanks Canis!) We also had glow-in-the-dark bracelets and a bunch of music-synced lighting on the tables and corners.

We also had a retail bar and were giving out free drinks to really anyone who talked to us. Although I did forget the secret word a few times which made for an awkward welcome!

"Please enjoy a free drink with the password....er...one sec brb.."

The real MVP of the party was DotOrNot, a DEFCON Goon who volunteered to DJ the party after our two DJs cancelled (right after his shift, too.) He was great, and I'm going to request him for next year. The feedback I heard was positive all around.

For next year, I'm also going to figure out how to make the phone booth happen up at the party. That, and more power. We had a whole party-line configuration ready to go where each phone at different tables could pick up and dial extensions and talk to other tables, but that got scrapped because we only had enough power outlets to plug in a couple of phones and the lights.

For a first year party, I thought it went great!

Brendan was able to meet Jack Rhysider, and even dragged him into the party briefly! You can even spot a Call Center Village badge in his DEFCON haul!

Final Thoughts

There were a few things that I noticed but haven't quite worked through my thoughts on yet, but that I did want to share.

• Many people I talked to assume call centers were bad. Like, scams.

There was another contest called the Scambait Village contest which was effectively hitting back at scam call centers. I had to regularly tell people we work with the good call centers! (And that good call centers exist!)

• If hackers hate AI, it doesn't show.

I expected there to be more anti-AI feedback but pretty much everyone I've talked to was interested in using AI in some form. The morality concerns about the environment, copyright laws, artists getting ripped off, big-tech monopolies...they just disappeared.

I feel like I finally have a real direction for Call Center Village contest where I can carve my own niche and still have it map back to my industry work. We'll be sharing all of our AI agent system prompts and settings as part of the Call Theory documentation and script library.

We'll be applying for Party Line and Call Center Village contest again next year! See you then?