It was many years ago now. I was working for a Fortune 500 company in the Information Security department. Most of my job revolved around policy developement and compliance. My main role was to make sure policy was up-to-date with current industry standards and interface with operations teams around the globe to make sure they understand how to implement and, if needed, make suggestions as to how they could. There were only two people on the team that had any UINX/Linux background, me and our Directory. Thus, if anything UNIX/Linux related came up, I was asked to assist in the investigation with our lead investigator giving me direction. I was not yet a certified forencis analysit as it was not a major part of my job and I always had the assistance of Lori, our lead investigator. Unknown to any of us, that would change soon.
A call was received. The company had an employee that was leaving of their own accord. They were returning to a country having strained relations with the U.S. The caller grew concenred when they witnessed our principal, we shall call her Ellen, printing documents that may contain proprietary information (PI). As the company was a major R&D and chip manufacturer, it was of great concern to our physical security department and human resources as well. The future of our company depended on the research peformed to invoate new products. This was a high priorety, and though I did not know it at the time, a high profile case.
I was tasked with retrieving and analysing the UNIX/Linux based evidence. It was not certain yet if we had an incident. The first step is to confirm that the events reported did, in fact, happen. I closed the door to our secured workspace and headed back to my cubicle within. I picked up the phone and called Joe. While I do not classify Joe as a friend, we were good acquaintances from a time when we worked together in the same department. "Joe, can you talk freely? OK, give me a call back when you are alone. Yes, time is of the essance here." Joe is a good man. He kept me on the phone while he asked his guest to leave and close the door to his office. In hushed tones so as not to be overheard on my end or Joe's, I explained that we have a probable incident that I needed to confirm. "Is it serious?" Joe asked plainly. I replied in a rather deadpan voice " yeah, if this pans out then it could be serious." I asked Joe for thirty day access to the network and that I needed root level permissions on a set of computers and data storage for the RF group. "I will let you know if I need any more time than that." I insisted that he set my account to automatically disable at the end of thirty days so that neither of us forgot about it. As an investigator, we want to be above reproach. I was granted access to Ellen's UNIX workstation, and the group design and simulation computers, as well at root level access to the network storage systems. Joe called back to let me know my account was once again active for the requested resources. I swallowed as a stared at the screen. How do I even begin? There is so much data. I felt a bit of a knot growing in my stomach. "Begin and the begining and go on till you come to the end: then stop" said the King, -Lewis Carrol Alice in Wonderland. I started with Ellen's UNIX workstation. Making a disk image (DD) of the workstation harddrive. I then moved to making a copy of the home directory; making both a disk image and much more manageable backup copy (tar ball). I copied these to my Solaris workstation in a protected partition where only I had permissions. I further secured my workstation by removing or locking accounts that were not absolutely necessary. At this point, only my directory and Lori had access to my workstation. Only I had access to the protected partition. I then acquired the print logs from the UNIX print server. Calling my good friend Larry, I asked for the Windows™ print logs as well. I verified with Tony that I had the most recent VPN logs. I was set to begin my initial analysis. Why, then, did I still have a queasy feeling in my stomach. I was drained of energy. At this point I only wanted to sleep. I wasn't even finish the my first day of the investigation yet. This is going to be a burtal case, I thought to myself.
Day two, the analysis begins. I search the application logs, temporary files in Ellen's home directory and the trashcan. I confirmed my finding by reviewing the print logs. Houston, we have an incident. I reported to Lori and my director that, yes Ellen did print documents but that was all I could confirm at the time. Physical Security was called in. The director of our Physical Security Department, Jim, as a former FBI agent. We laid out what we knew to date. After some further analsys by both Lori and I, we reported our finding to Jim. At a level well above my head the decision was made to inform law enforcement. The FBI was called in to hear our case.
I was late to the first meeting with top level managment and the FBI. I already had something schedule, I no
longer remember what. I was on my way home, I think, when I got a call from the conference room. I told my directory
that I would be there soon as I can but I had to take care of some scheduled business. A strange voice came over the
phone. He introduced himself, I will call him Agent Smith, and asked me some questions while I was driving. Some of it was quite personal. I asked
in the most jovial voice I could muster knowing I was late for a meeting, is this a phishing scam in an atempt to get
my personal information? There were some uneasy laughs in the background. I could hear Lori and my director. I
immediately provided the information and Agt. Smith stated that he will run a background check.
It took quite a while to get all the information we needed. I performed my normal job duties during the day, then go
home and take a nap. Overnight I logged in remotely to the office and performed my investigative duties. The image
depicted here I took from one of those late night sessions. This is my Beagle, Kirby. I got up for a drink of water
and he jumped in my seat behind my laptop. I thought it made a good picture. My Beagle working dilligently behind
computer. I entitled it "On the Internet no one knows you are a dog." In a nod to the New York Times
The time pulling double duty was taking its toll. I was tired and having trouble concentrating at work. I was not able to spend my daily time at the gym because I needed the nap. Food was often quick if not nutritious. A stakeout was proposed. The FBI would wait for Ellen outside the office. Jim from Physical Security would be on hand. It was late. The parking lot was dark, just dimly lit by the parking lot lights. A large planter cast a shadow across the sidewalk adding to the darkness. Ellen was working late. She was finally on her way out of the door. She was stopped and searched. Ellen had with her documents that were labled proprietary information. We now had enough for a search warrent.
A warrent was issued for Ellen's home and computers belonging to her as well as the family. Copies of the disk drives were made and Lori performed the forensic analysis of those. It was determined that digital copies of the data were made to USB thumb-drives and by burning to CD/DVD ROM. Another search of the house was made. There was no evidence of digital media in the home. Were they sent on ahead? Were they even real? So many questions were left unanswered. Lori found more company documents on the private laptop. These documents were marked proprietary information. Now we found evidence in the home and on personally owned devices.
An arrent warrent was requested. Ellen was already on the way to the airport with her husband and child. We knew her flight information. She was leaving the country tonight. We did not have the warrent in hand. We had to wait on a judge to sign off. The FBI was at the airport monitoring the situation. She passed through security check and was on her way to the terminal gate. Would we get there in time. The FBI approached Ellen at the termal gate. Trying to convince her not to board the plane. She knew they couldn't touch her without a warrent. She refused their pleadings. The plane was boarding. The FBI was stalling. Ellen rebuffed them and boarded the plane. The warrent was too late. She was on the plane and it was about to take off to lands far away. We missed our opportunity. My heart sunk. Was all this hard work, 80 hour weeks, for the last couple of months for nothing? Is there any hope of catching her on a connecting flight? Did we lose?
Time passed. I did get my forensic analsys certification from the SANS Institue. I did not have any more interesting cases such as this. It was quite boring really. I left the information security team to help the R&D department with policy implementation full time. I was working with Joe again. Still, not really friends but good acquaintances. Joe asked about the case. I let him down gentley. It was during my time working with the R&D team when I was laid-off. One of the departments was not selling product and the company chose to include IT staff with the lay-off. I left the company. I got married. Eight years after the investigation I received a letter for the Depart of Justice.
I got a call from the local Dallas FBI office. Ellen was stopped at the border trying to enter North Korea. Apparently she was red-flagged for travel. Ellen was taken into custody and sent to the USA to stand trial. Over the course of several months I got letters from the Department of Justice, black SUVs parked outside my home and finally one agent asked if I would testify. I agreed. I was told that I would be served with a subpoena to make it all official. The FBI agent wanted to serve it to me at work. I told him that we needed to meet outside of work and picked a restaurant close by. I informed the agents that I worked in the security operations center (SOC) and getting in is a pain. This was just easier. We had a nice lunch and I took home my subpoena. Over the next few weeks I spent many hours working with prosecutors, agents, etc. I often wondered what my new wife thought of all this activity. What did my neighbors think with all the black SUVs outside my house nearly every night. What did the postman think will all the corospondence from the Deparment of Justice? The trial began. I took the light-rail downtown to the federal courthouse. The trial lasted but five days, begining to verdict. My former directory and Lori gave testomy the first morning. I was called in after lunch that first day and told to return on day two. In total, I gave nearly a day-and-a-half testomy. There were three days of deliberation. The verdict came back, "not guilty". News articles reported the inablility for procescution to prove intent. The entire case was built on intent. Ellen claimed no such intent. I have no reason to doubt her assertion. I believe that God ensured justice was complete. I believe this verdict was in answer to my prayers at the begining of the trial -- that she not be punish if she was not guilty of selling secrets to our competators.
My early career was working for a major semi-conductor company. I worked for many different groups while there. I supported manufacturing, test, engineering design, corporate IT, and information security. My career there spanned nearly 25 years if I count my earliest years as a contractor. Supporting manufacturing I learned that cost savings is king and many times management cannot be convinced with logic. This is one of those cases but I prevailed with a no-cost solution to a problem raised to me.
A manufacturing floor engineer called early one morning. He was breathless and I could tell that he was exasperated by something. The night before his test system failed and no one notified him. He was now a day behind and frustrated by the lack of action. I was new to the group and Larry knew that. Larry was normally a mild-mannered gentleman. I am sure his traditional Chinese upbringing had a lot to do with that. "My system failed again last night. Isn't there someway we can have a high availability system on a brand-new manufacturing floor?" Larry continued, "Nobody seems to want to help me. Can you find a solution?" In fact, I could. I told him of our standard solution. It was design specifically for situations like these. It cost $5,000 per node. Larry would take that offer to his management.
A little about the system Larry was managing. The test equipment was supported by two Sun Solaris workstations. They both connected to a common network attached storage system and could read and write data. One system was the primary, controlling the test equipment. The other system was running idle in case of failure. What we refer to as a warm spare. The test equipment would only talk to the Sun workstation with the primary IP address. If the primary failed, the engineer requested an IP address swap from our networking group. During normal business hours this could be a quick swap. After hours, though, required an on-call network engineer to be notified by a ticketing system, make a VPN connecting into the office and submit the changes requested. This could take an hour or more. We were told that an offline piece of equipment cost the company $100,000 an hour in lost productivity. You can see why manufacturing engineers took this seriously.
Larry's management came back. They did not want to spend the money on the commercial high-available software. It is not an expenditure; it is a cost savings. I asked, "Larry, how many times per year do you have a system failure?" His answer was a surprise low number, about four. If we use our number of $100,000/hr of lost productivity, then four times a year was $1 billion in savings if we assume just one hour of down time per incident. Larry took the numbers to his management. They were adamant; they did not want to pay for high-availability software. Larry inhaled sharply and blew it out all at once. "Isn't there something that anyone can do?" I asked Larry to let me think on it for a bit. I would get back to him with an answer.
From an early age I looked at life differently. I credit my interest in photography for some of that. There is more than one side to any subject and I had to find the other side of this one. Upon confirming the version of the operating system on his test equipment with Larry, I set upon a solution. I did not know if it would work but it had to. I built a test environment of Linux operating systems. Yes, I know the test equipment is using Sun Solaris. The Linux workstation had similar capabilities and was a free solution for a test environment. I read up on the virtual interface capabilities of both operating systems. They were very similar with only minor syntactical changes. I developed my test environment and debuted it to Larry. He was impressed. Speaking more rapidly than I ever hear him, Larry wanted to know when he could put this into production. I asked him to give me a week. I needed to create an alerting mechanism so he would know when a failure occurred and not rely on the manufacturing floor personnel. This allowed him to check up on the system to make sure the fail-over took place without incident.
The system was elegant in its solution. I started with the desired results. We wanted a system that could switch from the primary workstation to the warm spare system upon failure. We wanted this to occur automatically. We did want to rely on a person, be it manufacturing personnel or networking engineer to perform any task. The solution must be self-contained. Next we listed what we knew about the system. The test equipment was controlled by two Sun Solaris workstations. The test equipment could only communicate with one system at a time. The test equipment could only communicate by IP address; it did not have Dynamic Name Server (DNS) lookup server capabilities. Finally, we could readily configure the test equipment. My solution was to use three IP addresses: one, on which the test equipment would communicate, and two for the main interfaces of the controller workstations. I used simple scripts to test the communication between the two controllers. The secondary controller tests the network connectivity of the primary every minute. If the primary fails to respond, the secondary starts the process of configuring a virtual interface with the communication IP address. The test equipment should not ever know the initial-primary controller ever failed and just start communicated with the new controller. An email is then sent to Larry to let him know of the failure so that he may verify the change over. On startup, each system checks to see if the communication address is in use. If it is not yet in use, the first controller to boot configures the virtual interface and become the primary. It is not perfect but it is elegant in its solution
I performed the entire task without setting foot on the manufacturing floor. I never laid physical hands on the controller or test equipment. The entire process for me took place in my cubicle. One morning Larry was at our location for a meeting. He stopped by to introduce himself. When he entered my cubicle, he chuckled. I turned to ask what was so funny. He pointed to an open box I kept on my shelf. Within that box was a piece of paper upon which I printed, "Don't think in here!" Now that box had multiple meaning to me. The one most relevant to this occasion was don't settle for the standard answer. Look for innovative solutions. You just might be surprised.
It was the early 2000s and it was not since I joined the company's Information Security Team. I already gained a reputation at the corporate IT level for getting things done. I did not play politics and truely only wanted what was best for the company. Many of my coworkers had their own agendas. That is fine, but all request for production had to go through me for InfoSec approval; and I asked a lot of questions.
My director asked me to take over a project from one of my fellow InfoSec team mates. This is not an unusual request, we traded projects often. It sometimes helps to get a fresh set of eyes on a project to help move it forward. This project was to replace the current Windows™ client firewall software. Steve introducted me to the client firewall team when I attended the first meeting. He informed them that I was taking over the project from an InfoSec standpoint. The client firewall team wanted to jump right into work but I put the breaks on. I needed to read the charter as given by the VP of IT Operations and review what took place to date. We would start anew the next week once I read the documents. Speaking quickly and abruptly on member questioned my motives. I replied using my best "nighttime FM DJ voice" that my motives were to do what was best for the company and based on the VP directive. I then asked wryly, what her's were. On that note I left the conference room.
The next week I walked in the conference room prepared having read all the plans to date. It was my opinon that if we just upgraded the existing software in place that we could achieve the desired outcome. It was pointed out by Randy, the project IT leader that was not an option. His voice started strong and then became quiet. Given Randy's tone I guessed that he agreed with me. Our VP's written instructions were to replace the existing software. It was too bad. The updated software was much more capable than the eventual replacement. It performed exactly as desired and was centrally managed. That last part was the issue with the version that we were currently running. The old, unsupported version was not centrally managable. We deliberated as a team. Waying the advantages and disadvantages to each solution. I could tell from the tone of the room that we all understood the best solution was not on the table but we could only pick from what was available to us. It only took us a matter of weeks to pick a candidate for evaluation. Randy from the IT Operations team built the managment server and we deployed the client to the client firewall team's computers. It was decided that the interface was not great but the solution met most of the requirements set about by the team. It was agreed by all that Randy would present our findings to our VP.
I was broadsided by Steve when I got back the office. He was speaking agressively, and rather loudly. How could I have allowed that product to be approve? At this point, I knew the bottle-neck was Steve and not any issue with the rest of the team. I tried to explain as calmly as I could that this was the best solution of our given options. Steve was adiment about upgrading the existing software and was not going to take no for an answer. I simply informed him that was not an option given the directive from our VP. Steve had some unkind words about our VP and the directive. While I agree with his sentiment, I had to move us forward in the project and not joust at windmills.It was about this time that my director, Brian, walked in. I had the cubile next to Brian's highwall office. Inquisitevely Brian asked what was going on. I allowed Steve to state his possion. The words rushed out of his mouth like water cascading over a waterfall. His voice still elevated like he just walked out of a rock concert and was still struggling to hear his own voice. Brian gently close the door so we could talk. Steve laid out his objections point by point; filled with emotion. I get it, Steve was not pleased with the decision. Brian then asked for my point of view. It was simple. Our VP clearly stated to replace our current client firewall. Upgrading it in place was not an option. We reviewed the remaining options and chose the one that most closely aligned with InfoSec goals and was centrally managable. Brian quipped that it seemed like a done deal. Steve stormed out of the office. I think he felt betrayed. I had a job to do, I simply did my job.
From the onset of taking over this project, it was my goal to make sure that we were successful. I told Brian when he asked me to take it on that failure was not an option. This was the attitude that I started my career with when I left home after college. My parent told me as I was walking out the door, headed for the big city, that the door was always open if things didn't work out. I simply stated "failure is not an option." I heard my Dad snicker under his breath. Having just finished his book I understand now why he thought that was funny. I, afterall, was my father's son.
I never liked school. From a young age we moved quite frequently. This made it difficult for me to establish long term relationships. Looking back on it, I gave up on that a long time ago. I was satisfied with new short term friends that I would abandon with our next move. It was just a fact of life for me. I also had undiagnosed ADHD and Dyslexia. At least I was never told I had these problems. I just struggled through. I thought everyone else was the same. Finally, I was always a small boy and often picked on. The combination just made school an uncomfortable experience for me.
In elementary school I taught myself to deal with the distractions of the classroom. I had a technique that I developed to block out the sounds and movements of the other kids. In the third grade I was analyzed by a doctor at the school’s insistence. They checked my eyes, hearing, and various other things. The only thing that I took away from all the poking and prodding, I had a slight curvature in my spine. Great, do I have scoliosis, too? Nothing changed. I was allowed to return to class. For all of that, I was held back and not allowed to advance with my classmates. Did I tell you I did not like school?
I started my Middle School career at Estee Middle School in New York. I tell you, my first year was uneventful. I made no connection with anyone. I showed up, did my time and left. I was almost like a prison sentence. My second year, grade seven, was much better. This despite the fact that I knew my family was moving again. I had no particulars just that we were moving once my father’s transfer was approved. There was a group of boys which whom I started hanging around: Steven, Doug, Michael, and Bruce. Later Aaron joined the group as well. I loved my new friends; cherished them even. We had many of the same classes together, including Phys. Ed. and shop. Alas, the time came to move and we traveled from New York to rural Texas. I tell you, what a change. I was still picked on but I did make some new friends: Rex and Roy (the twins), Paul, Randy, William, just to name a few. These friends was mostly able to carry on to college.
High school was uneventful all throughout. My friends I made in middle school were still my friends, well except for William who moved away. I started working for a fast food chain in my 10th grade year. I did not realize it until much later but I was not picked on in high school I never did figure out what changed. I worked all the way up to graduation. Even after being held up at gunpoint. I struggled through to make grades but I did manage to make it all the way through school.
I was off to college. My older brother, Grant, asked me what I was studying. Computer Science, I replied. I quote: "You are going to study computer science with your ADHD!?!" Shocked I proclaimed that I did not have ADHD and my mom laughed. Not just a little. No one told me. I just assume everyone was like me. We were kids, we had excess energy. That was normal. Apparently, I was well above average. I was taking an accounting class taught by a local CPA subbing in because one of the professors was on leave. I did not do great but I got solid Bs and Cs. I was happy with that. My teacher approached me one day and asked if I was ever diagnosed. I had no idea what she meant. She explained to me the mistakes that I was making on the homework assignments and thought that I might be Dyslexic. No, I was never diagnosed; or tested for that matter. She made an appointment with the university resource office to have me tested. Great! Not only did they confirm my ADHD but now I have Dyslexia, too? Well, at least that explains the numerical transpositions. Reflecting on my educational life, it explains a hell-of-a-lot more. Well, at least I have an answer to my difficulties. Now that I was aware, I could compensate. Why does life have to be so hard. I was able to graduate college with a 3.8GPA. Not bad for someone with new found challenges.
I struggled more than most for the same results, well not even as good most of the time. I had trouble reading. That was a big problem for a university education. I had trouble with spelling. I love computers and autocorrect. I refused to allow my challenges hold me back. I stated in another blog that failure was not an option. This is a long standing mantra to myself. Adapt and overcome was another mantra. Always moving forward to achieve my goal. I graduated thought I never became a programmer. My field of choice was system administration. I did it well for many years. With hard work, determination, and the ability to learn, not just classroom but myself, I was able to succeed. I have no special talents. Anyone can do what I accomplished. I quite often make it my mission help those coming up behind me see their value, see their worth, and know that they can make it too.
All managers were away at an off site meeting. Meaning they were playing golf. If this were any average day then that would be fine. This did not turn out to be an average day. With no management on site and no senior analyst willing to make the bid decisions, I stepped up. Wrong or right, I did what I thought was best for the company and the data we steward.
On this day we had a power outage. We had multiple power feeds to the facility and we had battery backups to carry us over until the secondary feed kicked in. Today we had multiple failures. Our primary feed was interrupted. At first we did not think anything of it. We waited to switch to the secondary feed. After a bit, about five minutes I estimate, Carl checked on the data of the switch over. I do not know what drove him to do so. I am glad he did. I followed Carl to the front entrance of the data center. Carl went into the battery room. "Well, what does it look like?" I asked. We never switched over the secondary feed and Carl was unsuccessful at getting it to switch manually. We officially had a double failure.
Standing there in front of the data center, we discussed our options. Carl informed that the battery backup was rated at thirty-minutes. Great, now we have a starting point. The power feed was out about five minutes to this point. I remember turning to Chris, technically he was more senior that I, and asked what we should do. Chris took chart of trying to contact our management. I devised a backup plan as I suspected that management would not answer the phone on the golf course. I was right. On the assumption that we had thirty total minutes of battery time and that we used about five of those already, I decided that we would start to shutdown the data center in about fifteen minutes unless power was restored. Our clock ran out. Fifteen minutes was up. I instructed Chris to have the other start shutting down computers in the data center to preserve the data. He refused. I, however, executed my plan on those systems for which I was responsible. I was able to shutdown all but two of the file servers. Twenty-three of 25 ain't bad. I ran out of time. Without knowing it, we over loaded the capacity of the batteries to maintain the data center for a full thirty minutes and I ran out of power.
City power was eventually restored and the data center was returned to production. In the shutdown, I only lost two disk drives. With the RAID systems in place that means there was no data loss. I replaced the disks from stock and called our supplier for an RMA number. Management returned sometime after lunch. I was in my office working on the RMA and writing up my notes to present to Gale, my department manager. I was not among those who swarmed management in a flurry to tell them what happened. About three o'clock that afternoon Gale called me into his office. I asked for a second as I pressed the enter key to send him my report on the event as I saw it. "I am on my way."
When I got to Gale's office, I told him I sent him a report on the incident in an email. He took a moment to read and asked me a few questions. Who told me the batter was rated for thirty minutes? Who made the decision to finally shutdown the data center? When did we know that secondary feed did not switch over? What did I pick fifteen minutes? That last one I had to answer sheepishly. I felt that I needed ten minutes to shutdown all the file servers cleanly and I took a SWAG (some wild ass guess). Gale's response was that I did the right thing for the right reasons. He thanked me for taking charge. He asked why I did not wait to hear back from the management team. I simply informed him that I knew they were on the golf course. I play golf. I know that it is bad form to leave your ringer on when out on the course. That I felt the likelihood of contacting management in time for a meaningful response was near zero. Was I wrong? Feeling put on the spot, Gale simply answered, "no."
Those of use with ADHD are uniquely qualified to handle crisis situations. We are wired for it. There are studies that show we are able to take in a lot of information from many sources and quickly digest it. Our creative mind kicks in to hypothesize solutions, and once decisions are made, our ability to hyperfocus on the task well past the point where mere mortals started to fade. We sometimes are afraid to act. We fear rejection. We often suffer from uncontrollable anxiety. I was able to control those negative emotions, make a decision, and execute despite objections from my peers. I did what I thought was right. I think that I was able to do this because I was grounded in my own belief system. I could stand on that and say, this is why, right or wrong. My ego is not tied to my beliefs so if I was wrong, I am able to change in light of new facts.
I worked for one of the big three anti-malware, anti-virus, solution providers. At the time I was on the customer helpline taking calls for those that needed assistance. This particular morning I got a call from a customer that needed help with their product. The name was similar to ours so I was busily trying to help him find a solution. In order to take this past the initial call, though, I had to open a case. I asked for the company name so that I could search the backend database. Alas, I could not find the company name. I asked for his e-mail address so that I could search on that. Nothing. Finally, I asked the caller to provide me with the full product name for which he needed help. Oh, that is now our product. That is a competitor product. After a bit of going round about who he actually called, I pulled up the business support number for our competitor and provided to him. He was very thankful and I wished him a nice day.
My colleagues asked why I provided him with the information. My reasoning was simple. One day he may tire of our competitor and look for a new anti-malware vendor. I wanted to leave him with the impression that we are he to help. Most everyone disagreed with my assessment. They all made comments of how they would berate the caller. That is just not the company I work for, nor do I want to.
This is a bit of a personal event. It is in no way work related. I do feel that sometimes God put up barriers to see how we will respond. Do we respond in a way that pleases Him or do we please ourselves?
One morning I woke up late for work. This made me late for our weekly meeting and it threw off my entire day. I woke up late, that means a quick shower. I grabbed a quick cup of coffee as I ran out the door. Taking my normal path to work I found that the road was under construction. I turn to find a path around the construction zone. There, too, was more construction on my alternat path. Taking the detour because the road was blocked, I found another path to work. I used to live in this neighborhood so I knew it well. When I get to the next major street where I will need to decide to go north or south, there is more construction. I am forced back south to a busy artery. Merging back into traffic I sit and wait like everyone else during rush hour.
I turn the corner. I am finally on the last leg of my trip into the office. A drive that normally only takes me thirty minutes is dragger on closer to an hour. I am in the far-right lane as I intend to make a right turn soon. My frustration builds as I see a car stopped in the lane ahead of me. Snorting our a breath of exasperation and thinking what could go wrong next, I see a child pop its head up in the back seat. Now, rather than be exasperated, I change gears. This mother and her child are stalled in the right lane of a very busy road during rush hour traffic. Within a blink I made the decision to pull over and help. I pull into the parking lot. I approach the car and ask if she needs assistance. She states that her boyfriend is on the way. It is not safe for her and the baby to be stranded in the roadway so I explain that I am going to push her car into the parking lot. Put the car in neutral and I will push. As I walk to the back of the car I note the first person stop to help me. I quickly explain the situation and we push the car into the parking lot together. I wish her well as she points to a car pulling into the lot. That is her boyfriend.
I arrived late to work that day. I only told my boss that I overslept and traffic was bad. He understood and because this was not my normal, he said nothing more about it. Ultimately, I believe this was God’s lesson to me that morning. He put up the barriers so make sure that I remember what was important. I think ultimately I passed the test.
I was a senior system administrator or architect for a large, multi-national corporation. My director asked me to deploy NIS servers in India and Japan. The work would be completed remotely and I would not be on-site for the deployment. The project required new hardware and configuration of the operating systems.
I sketched out a project plan outline and then met with my director. I had questions. What was my timeline? Were we going to purchase the hardware and ship or was the hardware being purchased on site? Was I expected to configure the operating systems or would one of the junior admins? The question of the hardware was the toughest as he did not yet make that decision. We talked it over and decided it would be quicker if we purchased the hardware and shipped it.
The hardware arrived and the datacenter admin team contacted me regarding how this was to be deployed. I informed them this was going to be a configure and ship operation. I would do three systems at a time. They racked first three in the temp-racks. Loaded the operating system, and addressed them in the temporary address space we set aside for such things. I configured the Network Information Service (NIS) for each, configured the primary by name and IP address, and then configured the two backup servers. I then started the process of synchronizing the data to pre-populate the tables. After some testing of the primary and secondary servicers and services, I declared they were ready.
I contacted the system administration team in Japan to request three IP addresses with the names that I specified for each. The IP address range was to be for infrastructure only. Within one business day I had the three addresses, which I verified. I modified the configuration files with the new IP address information and then manually set the network configuration by command line. I tested the communications among the three systems to verify that all was working as expected. The datacenter team packed up the hardware and shipped it to Japan.
The hardware arrived in Japan. I got an email once it was installed. I scheduled a training session with the team to go over how to integrate into their environment. I set up a time from 10PM - 12AM my time to perform the training. This worked well for me as it did not interfere with my regular duties.
It was early in the 2000s. Secure authentication methods were standard in universities and businesses but for large corporations stuck in the 1960s, clear-text authentication methods like telnet and ftp still reigned supreme. It was the spring of that year. My director of Information Security asked the team if they had any projects to propose. The leadership team would meet that summer to discuss which priorities to fund. I suggested now was the time for our corporation to start using secure authentication methods.
I was tasked to put together a presentation of what I thought the migration might look like. During my presentation I got push-back from the networking team. Their stance was that since we were using switched networking, one could not see passwords transmitted across the network. My very loud guffaw filled the room. It was the only sound heard in the now quiet meeting room. Our disagreement ensued. My director and the Vice President of Information Technology quelled the room. At that point I stated that I could prove it. “When!?!” was all that Tony stated. I promptly replied I would do so in next week's meeting. With that, the meeting was adjourned.
At this point I set out to prove my point. I designed a test that included both corporate and personally owned devices. My natural inclination was to run my test on a Linux virtual machine. It was pointed out to me by Jorges that our VP felt that anything done on Linux requires high levels of expertise. With that in mind, I performed the test in both Windows and Linux. I had a Sun Solaris workstation that I could configure to accept telnet and ftp clear-text authentication. My standard configuration was to disable those weak protocols. This was my victim computer. My attacking computer was my personal laptop at the time named “DarkTower.” I installed publicly available software on my Windows operating system. I then installed VirtualBox, an opensource virtualization software. Here I created a Linux VM with more freely available software. I also installed Wireshark to capture the data packets during transmission. My lab environment was ready.
Connecting Darktower to a corporate network jack in another part of the building, I ran my first test from the Linux VM. A platform with which I am much more familiar. I started the Wireshark network analyzer, and then the ARP spoofing software. I simply entered the IP address of the victim machine and gave it a minute to start re-routing its network traffic through Darktower. I then simply made a telnet request to the victim computer. The Wireshark packet capture saw the clear-text authentication as one would expect from the same computer. Then, the ARP spoofing software displayed in the output window my username and password. There was no guessing. I performed the exact same test from the Windows side. Again, the ARP spoofing software displayed my username and password in plain text in the output box. No guessing; it was spelled out clearly.
I put my screen captures in to a slide deck for presentation purposes. I then printed three copies for handouts. I presented my findings at the next week's meeting. With my opening slide on the screen, I walked around the table and dropped my printed presentation in front of my VP, Director, and Tony from the network team. I quickly covered what I did, the results, and how it is detectable by the network team. At the conclusion of that meeting, my project was approved.
It sometimes takes extraordinary effort to move what set idle for a long time. To many of us today deprecating clear-text authentication for more secure methods is common sense. Those in power are quite often not tech-savvy and must explain in non-technical terms why we need the change. It does not hurt to bring pictures. Our new hires helped push this change as well. Coming straight out of university, many scoffed at the fact we were still using telnet. Even then I had to make concessions. We started with Internet facing endpoints and accounts with administrative permissions. I heard from one of our engineering design teams that once SSH was pushed out to all endpoints that new hires and power users asked to be allowed to start using it right away. There was pent up demand for the automation possibilities that SSH provided that were not available with other clear-text authentication methods. If you believe in something strongly enough, fight for it. You might be able to change the course of business.