I always am very pleased to see younger folks that both know HOW older technologies are supised to work, and care enough to do things correctly.

Thanks Sarah for showing a real time trouble shooting sequence really brought back gobs of old memories from the 1970's. I can't count the number of faults exactly like that, shorted and or bent terminals, or a solder ball sitting on the banjo that I have cleared in my 31 years with just a quick visual inspection. Many a time I came in at 4pm and was handed a trouble that they had been working on the day shift with no resolution and I cleared it in a few minutes as I always did that visual inspection prior to getting out the books and tools. Bent contacts in connectors was another quick fix that everyone had overlooked. On trunk troubles I would sort them out as to trouble types and work on only one type at a time and there were only a few causes to that type and I would clear bunches at a time because I was lazy and made it easy on myself. I was actually mediocre at best but got a good reputation by being sneaky and fixing the easy stuff first then had plenty of time to spend on the head scratchers. My eyesight was always lousy so I was crap on the backplane, especially the newer wire-wrapped backplanes. Looking at all those close wire-wrapped terminals gave me virago. We had guns that tightened up and clipped off the cable ties at a specific pressure so as to not damage the wires. I still use one they are not that expensive.

You know. In a hundred years time, in the year of 2123, there will be more working 200 year old windup grammophones around than working CRT-televisions, AM/FM radios, videorecorders and analog telephones from the second half of the 1900's. Thinking of you guys keeping generations of telephone centrals alive and working is therefore of utmost importance for our common technological history due to their incredible rarity.

A thermal camera might have helped locate the fault, if the short was causing the wiring and components to heat up -- but you'd probably consider that to be cheating!

Panel machines amaze me. I can still remember the very first time I walked into the PArkway central office in Seattle for a T-Carrier class and was met by the odd smell of hot grease and burnt cork. Smelled nothing like the Step-by-Step machine I worked. 😉Spent some time just watching the the machine work. Fascinating.

I manage a team of maintenance technicians in manufacturing. You would be amazed at how frequently we run into similar situations with lack of documentation and needing to troubleshoot hardware the way you had to in the video. Good job!

Hey Sarah, thanks for that! I enjoyed every minute, the complexity of central offices from back then never ceases to amaze me. Sheer engineering marvels and I wish I'd learned about this stuff 40 years ago. Cheers from the UK.

I have scars from reaching into server racks with zip-ties... One cut almost needed stitches. I like my Velcro straps, but eventually I would love to learn wax lacing. Great job as usual! It makes me happy seeing people take pride in a hobby, especially the electro-mechanical types of hobbies.

Great job of troubleshooting! @ 31:36 just before you said "send it around the back" , you needed to take that short tag end and tie a half hitch with the long lacing end to lock it in. I've done a good amount of lacing as an AT&T tech, and that half hitch will lock that loop in very well.

If you're worried about blowing a fuse, grab one of the 24v halogen clippin lights and use it as a fuse. Instead of consuming fuses you'll just turn the light on 😊

Wow, brings back loads of memories. The sound of the panel office…I’ve heard that spinning selector sound whenever something major happened in NYC (where I worked) An other sense that brings it all back is the smell….any time I’m near a box of crayons it smells like panel, because the wires and numerous other items were coated in paraffin. Thanks for the memories. 😊

That was an AWESOME troubleshooting and fixing video ! This is not only very interesting to watch but can help others in the future as a troubleshooting aid for the museum and for troubleshooting in general. The museum is SO much better with you there, Sarah ! Thank you !

It's amazing how similar the debugging process is to what I do today in web development. Tracing through what works, what doesn't, how far the signal gets, where it doesn't, or where it gets corrupted, until you find a single parenthesis out of place. xD I love watching you at work. Great fun!

Well done Sarah! As an electronics engineer, I can truly applaud your debugging. Had the short been a metallic insect, you would have clearly found and eradicated the BUG ;o)

Thanks Sarah. What a great video. You really laid out how to think and work like a veteran trouble shooter. I got a call one day from a guy who had been using an IBM PC XT for close to 20 years (yeah, I know). The PC had recently started acting like someone had tuned the big red power switch (remember those on IBM PCs?) off and quickly back on. The power supply appeared to have been undisturbed and the cover was installed with security Torx screws. With a little gentle rocking I heard something. I got my security Torx driver, took the cover off and found an extra cover screw lying between a heat sink and an inductor lead which had just a little solder tinning exposed above the circuit board. That screw had apparently been there since the power supply was manufactured and only recently made firm enough contact against the inductor lead to cause trouble. It is interesting how a latent short can sit around for years, in a panel frame, or PC power supply, and just the right temperature shift, vibration, or whatever, will finally push the short together… ensuring job security for a repair person.

There's a CO three blocks from my apartment that was built in the 1920's and still operates today. I can only dream of what kind of treasures are in that building (probably more internet stuff than POTS nowadays, so the cool stuff has long been silent), especially since South Central Bell bricked up all the windows sometime before I was born. I don't really know the story of how you guys got a CO and equipment to start a museum, but I would love to do something like that way down here in Alabama. Less than an hour away from my home in Birmingham is where the first ever 911 call was placed.

You couldn't find a bigger soldering iron?! 🤣 Thankyou Sarah for another amazing video ♥

Thanks, Sarah, for a very interesting video. You brought up an excellent point that troubleshooting becomes a lot more intense when customers are out of service. I enjoyed the wax string lacing demonstration. I have always been impressed how great a job the Western Electric installers did when lacing up the back plane wiring no matter the type of switch.

its neat how you can see problems happening in real time on these old mechanical switches. it almost throws a tantrum haha.

Wow. That system is quite a maze to work your way through. I think you’re one of the few people left in the world that understands it so well. I’m always impressed by how you figure things out. A great analyst!

All this fun stuff is making me want to buy another electro-mech pinball machine to restore. Another awesome video!

Great troubleshooting video! It also illustrates the relationship between schematics and wiring diagrams, and the immense amount of labor required to construct these electromechanical switches.

Lovely work lacing up the looms (probably not how you say it in US) :)

I remember the old timers they took good care of those switches and you didn't dare touch THEIR switches !

Awesome troubleshooting Sarah, and it your lacing was extremely impressive. I had learned how to lace back in the stone age, but have forgotten how to do it.

I used to work A-7s, it had a lot of relays and relay cards. The TA-7C almost exclusivity had relays and cards made in Iran. You couldn't get new after the Fall of the Shah. Please note, electromechanical relays do have a usable life, and those are teaching the end of theirs. The tantalum pads erode, corrode and Arc Over fusing the relay shut, the springs and leaves fracture and break leaving the circuit open. Everything ends.... everything.

I just wanted to let you know that I've been trying to find the time to offer my self to volunteer but due to mt own obligations, job and personal issues I've not ben able to do so Sarah. But seeing you and your passion in maintaining these systems bring me great joy and awaken my own passion and I hope to see more in the future! I hope to stop by again - Ryan

I can _smell_ that blueprint.

A housekeeping hint for all the cut cabling in the back of the panel. Lace them back into their own form then you can move them as a bundle. All the wires look to be color coded, but it appears that all the wires in the "Buss" are coded the same, or nearly. That is troubleshooting heck. The wires are bundled at the old Western Electric plant to aid in placing the laced loom in the frame. There was a wire board operator who spent his entire day running wire around a bunch of nails according his instructions. Then he would lace them all up and cut the wires. That whole assembly went to the operator that soldered all the terminals, and he would check his work with a "Beeper". Labor intensive but "Made in America".

I was working in the Panel office that served the downtown and outlying areas of Newark N.J. ie. It was a large office that covered 2 floors, I will never forget the deafening sound of that office on the afternoon of November 22,1963 when word got out that Kennedy was assassinated.

It's so cool to see a mechanical switching office still working. I worked in a DMS100 office for about 10 years but we recently got moved to our call centre office which is a standard cubicle farm so don't get to see the equipment anymore. I just monitor stuff so don't really work on it but it was fun to be in that building and see things get added, changed etc over time.

I love the in-depth videos like this, really shows a lot about how things were done and troubleshooted when these switches were still in use.

We would drive past an office like this when I was a kid. I peered in as best I could as we drove past-And I would see what looked like a warehouse with rack after rack of stuff like this. Who knew what marvels went on inside!

Very interesting Trouble and great troubleshooting, Sarah!

May fav part of this video was the cable lacing. Those zip ties were really bugging me. Cable lacing really is a lost art and I would love to see it used more often.

Love the analysis. There are so many tips in your videos. I’m building a step by step switch and this video just taught me how to lace properly. Thank you .. J

It’s nice to see you guys take care of this old equipment. Here in Norway the entire landline network is retired. The last subscribers were shut down last year. Tho the cables remain to supply other forms of communication for the time being (a few years). An era is over and it’s sad to see it go.

always a pleasure watching! congrats!

woah, i LOVE to see proper troubleshooting like this ... Sarah you are amazing

Well that was today's lesson in humility. I started as a hobbyist in electronics at age 3.5, I am now in my late 30s. I spent +13 years working as a research engineer. I had no idea a special cutter existed for cable ties until now. 1. Thank you. 2. How is this tool for such a common use so obscure? I don't even know the name of it.

Wow what an intense debugging job! Great technique to chase down the problem though. I had not realized how intense and concentrated the wiring behind the frame is, ouch!

Great demonstration of lacing. Thanks for showing that. We sold spools of the waxed lacing tape at Vetco but I never got to see how it was done.

Great stuff. I remember looking at those banks of cables as a trainee thinking wtf. And the noise!

That is why it is necessary to wrap the place of soldering in an insulating tube so that it does not close with adjacent contacts.

MEGA huge soldering iron, I love it

Awesome you answered some of my questions.

Reminds me of the wall of relays at my last job.. A 1950s nurse call system in a hospital. The coils had a light wired across them so you would know which relay had power. Chasing down glitches could be a nightmare.. That looks to be orders of magnitude more complicated than the nurse call system.

If someone were to place a call when this fault was happening, what would happen to their call? Hangs up? Some sort of busy signal? Stuck? Also why not trace that abandoned wire bundle and eliminate stuff like that? Is there a concern that moving it around too much may cause other faults?

The wiring job is super neat!

Nice troubleshooting, and good eye. (Also nice to see someone who still knows how to lace properly!)

It would have been good to actually busy out the E decoder connector and then show that the call sim could work normally with just one bad component busied out. A lot of work lacing wires that don't actually go anywhere just to make it look good. Kudos!

Good to see you again Sarah

Great video Sarah

You don't need a special tool to cut zip ties properly, small cutters as you were using to remove them will work just fine. My personal pet peeve is when they cut them on an angle leaving a sharp point just waiting to rip the back of my hand open.

Great video! Do you never use a tone sender/tracer when trying to find a wire run? Or was that not possible because of the wire connected to ground?

Really interesting to watch. and think about when they were used in larger settings to be a tech and get that work order. i don't think they had a few hours to solve the problem. zip ties are great but when they are cut to short you get that minefield of sharp edges. so nice to see that cable stitching being done and very important to do.

Enjoyed very much.

I love this stuff. We lost the character of the phone system when we went digital. I wish this weren't on the other side of the country.

The switch sounded like it was full of angry bees at the beginning when it couldn't find a sender. Nice video! (I am an old Nortel guy who cut my teeth on a SP-1)

12:50 Wow, a schematic printed in cyanotype? A literal blueprint for the wiring

Fascinating. The largest relay I've had trouble with was a six-pole double-throw in a National NCX-5 transceiver. The engineers really pushed way past the specs, and they were switching 280 volts DC with contacts rated at 24 VDC. To make it worse they were switching 280 volts off with the break contacts and turning ground on with the make contacts, so a few milliseconds of time they were shorting. They didn't last more than a year or two at best. Ma Bell probably didn't push things that far. I wonder if you could have used a current-clamp type of meter to find the short? Or do you only use period-correct test equipment?

Happy late birthday panel. Also. A fellow trans girl being ideological about safety precautions and protocols will never not be cute and awesome.

Yeah, funnily enough the same zip tie issue from the old times translates into current times. I just so happened to slice my whole hand open on goddamn cableties in a data center

There are two main faults in electrical engineering: either there is a contact where it should not be, or there is no contact where it should be)

Nice to see someone who knows how to lace cable. But, I am one of the people who uses cable ties.. but I do have the proper tool and use it every day :-)

Brilliant as ever! I have just said to my other half: In 2024 I am travelling to visit the Connections Museum. He did not say: No. Incidentally I love they way you (and others from USA) say “sodder” rather than “solder”; in England it sounds really funny and somehow a bit naughty😉. I do hope you will be there still next year!

As a Telcom engineer calling e1 and b channel/sigtran signaling ancient the marvel of mechanical switches is quite awesome! Then again I also very much like the old skool pinball machines :)

Total respect for your attention to nicely lacing the wiring back together at the end. I'm curious what kind of string are you using? Thank you for the great video!

Man I was your place like this existed around where I live

In wiring pipe organ relays I have a saying, "Neatness Counts". It counts in telephone relays as well. Lacing in fun.

Zip-ties are the devil, but when I have to use them, i've found that toenail clippers are a good inexpensive substitute for the special zip tie flush cutter!

super job

great job in fault reconciling in the code decoder

Great vid for FIRST TIME viewer! Now I gotta learn MORE antique electronics, thanks MA BELL! Just learned how to brew a tetrode from scratch and how to make ic's from sand... Okay, last was not real. Just saying...

I do wonder, why keep the extra wires? Do they serve a use just being there? Purely for the look?

this level of true-tech-mystery is addictive

Amazing old machines... but I couldn't help but think when you said that magnet was about 30-35 degrees C... imagine the old guys that worked in telco back in the day would have hurled some verbal abuse at anyone heard using such non-patriotic units of measure!😂

Somebody in your tour bumped their butt up against the rack. That is the initial source of the problem.

I have a wonder for you @Connections Museum. When i search for foults in our relayboxes for railway, we have two ways to do. With a volt meter och a lamp with a magnet for the ground and the test lead. Is it not possible to do someting sumular in your applications?

I used to be able to make that happen at my local CO back in the days of switched systems! I would take the CO down for a day or two before they could fix the problem.

I always cut my cable ty's at a 45% angle lol. I'm a retired tech. 41 years of doing that. People will be bleeding for decades. lol

Sarah, since you know one end of the offending wire is one side of the SG magnet, why not just pick a working one, connect one side of the SG magnet to the meter, and just scan for continuity on the SF contacts? That would give you one side and observation of the SF contacts should show you the other side.

Thanks Sarah, for another excellent video. It was not clear why you would leave all those unused spare wires in that cable bundle. Was this wiring left from some prior configuration? I would have pruned much of that out, since the bundle was open. However, that may be easier said than done.

Could you continuity check to ground to figure out which wires might need to be removed?

I like those eyeglass frames.

17:25 feels like a far too crude way to search for connections between 48 volts and ground. Discharge machining the contacts is often not a good way to make things last. I would have a suitable resistor, a small bridge rectifier, and an LED. Would light up visibly when in circuit, but not cause needless sparks. (A suitable resistor would be around 10K ohm or so.) edit: 24:30 is something I thought were a joke. That is a ridiculously large iron for the job. Heating up surrounding wires is a wonderful way to degrade the insulation and cause future shorts as the degraded isolation fails. I understand trying to be authentic to the period and such. But when it comes to maintaining the machines, it is better to use working practices that helps preserve the machine. 26:08 yes, those are improperly installed cable ties. They should be flush cut such that they won't cut someone's arm open in the future. (I know of companies that has fired people for not doing this properly. It is a liability for the company to ship products that can inflict such preventable injuries.) 26:22 "Does anybody ever do that?" Well, I do. To the point that I fix those that aren't done properly when I see them. 28:50 is a classic. Stiff wires getting pulled about over the years usually dislodges terminal blocks and other stuff over time. Now these days the issue is often cracked solder joints leading to intermittent connections, rather than bent metal terminals shorting out. But the root cause is the same, stiff wires getting forced about through years of service and eventually pulling something along with them..

Do you have one of those RF wire trace and probe devices? They could make it possible to trace wires without cutting the bundles.

What would you say to a video answering a fuse alarm, from a major and/or minor initiation to possibly showing the alarm bay to zeroing in by way of aisle lights and panel lights Possibly other types of alarms. This small but very effective tool would demonstrate the conviction to service from the Bell System.

Woah... I've always got the probes out, but I work on 5V digital stuff... I don't get sparks... I'm not sure if I'm jealous or glad. ;)

Sarah, It is a shame you only have 1 Decoder.. That would really pin point it. Stop blowing fuses HeHe. Sure, if the Sender did not release the connector there goes the Decoder. Good technique. Oh god memories back to 1970.

Now that you have discovered the terminals that provide that ground, do you go back and note that on the schematic?

Hey so I'm have not watched all ur videos yet but I watched enough to understand that these are real calls. I would love to see a specific video on where are these phone calls coming from and how they work with modern centers as a assume they are connecting digitally but I don't know for sure

Tnx, 🇵🇱 Warsaw .

The cake is a lie.. 😋

I wonder if a thermal camera could have been useful, following the one wire carrying current for way longer than normal? The newer higher resolution thermal imagers are getting cheaper and cheaper these days, and have enough detail to be pretty useful.

Is there a advantage (other than cost) of stitching over velcro-type ties?

Busy plug has what component(s) in it? How does it make the piece of equipment unavailable?

i think we should all go back to lacing, it's really not that hard honestly, and i agree. Cable ties are just horrible, one-use trash. (Except re-usable ones, but they always leave a long tail if you choose not to clip them for that reason)

BTW, we would never put all that back together till we verified we had completed the repair by testing. Yes, I know you are confident you found a problem, still there could, for example, be more to the problem.

I'm wondering why they didn't use a toner and tracer to find out more about the wire routing. It's a great tool for just that purpose. Also, why tie the unconnected wires back? It seems to me that it would be better to tie them to the vertical run where they are out of the way and not blocking the view of all those soldered joints. Or else cut them shorter, since they will never be used again. They emphasized looking neat, well removing unneeded wiring makes it even neater. Thanks for the informative video. 👍