Blog Shorts

Blog Index

2022

2023

International Corporate Espienage (2 Jan 2023)

It was many years ago now. I was working for a Fortune 500 company in the Information Security department. Most of my job revolved around policy developement and compliance. My main role was to make sure policy was up-to-date with current industry standards and interface with operations teams around the globe to make sure they understand how to implement and, if needed, make suggestions as to how they could. There were only two people on the team that had any UINX/Linux background, me and our Directory. Thus, if anything UNIX/Linux related came up, I was asked to assist in the investigation with our lead investigator giving me direction. I was not yet a certified forencis analysit as it was not a major part of my job and I always had the assistance of Lori, our lead investigator. Unknown to any of us, that would change soon.

A call was received. The company had an employee that was leaving of their own accord. They were returning to a country having strained relations with the U.S. The caller grew concenred when they witnessed our principal, we shall call her Ellen, printing documents that may contain proprietary information (PI). As the company was a major R&D and chip manufacturer, it was of great concern to our physical security department and human resources as well. The future of our company depended on the research peformed to invoate new products. This was a high priorety, and though I did not know it at the time, a high profile case.

I was tasked with retrieving and analysing the UNIX/Linux based evidence. It was not certain yet if we had an incident. The first step is to confirm that the events reported did, in fact, happen. I closed the door to our secured workspace and headed back to my cubicle within. I picked up the phone and called Joe. While I do not classify Joe as a friend, we were good acquaintances from a time when we worked together in the same department. "Joe, can you talk freely? OK, give me a call back when you are alone. Yes, time is of the essance here." Joe is a good man. He kept me on the phone while he asked his guest to leave and close the door to his office. In hushed tones so as not to be overheard on my end or Joe's, I explained that we have a probable incident that I needed to confirm. "Is it serious?" Joe asked plainly. I replied in a rather deadpan voice " yeah, if this pans out then it could be serious." I asked Joe for thirty day access to the network and that I needed root level permissions on a set of computers and data storage for the RF group. "I will let you know if I need any more time than that." I insisted that he set my account to automatically disable at the end of thirty days so that neither of us forgot about it. As an investigator, we want to be above reproach. I was granted access to Ellen's UNIX workstation, and the group design and simulation computers, as well at root level access to the network storage systems. Joe called back to let me know my account was once again active for the requested resources. I swallowed as a stared at the screen. How do I even begin? There is so much data. I felt a bit of a knot growing in my stomach. "Begin and the begining and go on till you come to the end: then stop" said the King, -Lewis Carrol Alice in Wonderland. I started with Ellen's UNIX workstation. Making a disk image (DD) of the workstation harddrive. I then moved to making a copy of the home directory; making both a disk image and much more manageable backup copy (tar ball). I copied these to my Solaris workstation in a protected partition where only I had permissions. I further secured my workstation by removing or locking accounts that were not absolutely necessary. At this point, only my directory and Lori had access to my workstation. Only I had access to the protected partition. I then acquired the print logs from the UNIX print server. Calling my good friend Larry, I asked for the Windows™ print logs as well. I verified with Tony that I had the most recent VPN logs. I was set to begin my initial analysis. Why, then, did I still have a queasy feeling in my stomach. I was drained of energy. At this point I only wanted to sleep. I wasn't even finish the my first day of the investigation yet. This is going to be a burtal case, I thought to myself.

Day two, the analysis begins. I search the application logs, temporary files in Ellen's home directory and the trashcan. I confirmed my finding by reviewing the print logs. Houston, we have an incident. I reported to Lori and my director that, yes Ellen did print documents but that was all I could confirm at the time. Physical Security was called in. The director of our Physical Security Department, Jim, as a former FBI agent. We laid out what we knew to date. After some further analsys by both Lori and I, we reported our finding to Jim. At a level well above my head the decision was made to inform law enforcement. The FBI was called in to hear our case.

I was late to the first meeting with top level managment and the FBI. I already had something schedule, I no longer remember what. I was on my way home, I think, when I got a call from the conference room. I told my directory that I would be there soon as I can but I had to take care of some scheduled business. A strange voice came over the phone. He introduced himself, I will call him Agent Smith, and asked me some questions while I was driving. Some of it was quite personal. I asked in the most jovial voice I could muster knowing I was late for a meeting, is this a phishing scam in an atempt to get my personal information? There were some uneasy laughs in the background. I could hear Lori and my director. I immediately provided the information and Agt. Smith stated that he will run a background check.



At the time this incident took place, I just filed my paperwork for a concealed hundgun license in The State of Texas. I passed my classroom and quarlifications. I was waiting on my background check to complete the paperwork. I guess there was no interference between the two background checks as I got my CHL late March while the investigation was still on-going.


There were many agents over the course of the investigation. Some understood computers, most did not. I tried to keep the terminology to a minimum when explaining what was found. Yes, Ellen did access those files. Yes, she did print out copies. No, I don't know what she did with them. I don't know if that is unusal for her, you would need to ask her manager if this type of activity is expected. I repeated that last one many times. I could not afford to make judgements. I was here to provide facts. It was up to others to judge. I did not want to be sitting on that jury.

It took quite a while to get all the information we needed. I performed my normal job duties during the day, then go home and take a nap. Overnight I logged in remotely to the office and performed my investigative duties. The image depicted here I took from one of those late night sessions. This is my Beagle, Kirby. I got up for a drink of water and he jumped in my seat behind my laptop. I thought it made a good picture. My Beagle working dilligently behind computer. I entitled it "On the Internet no one knows you are a dog." In a nod to the New York Times cartoon.
The time pulling double duty was taking its toll. I was tired and having trouble concentrating at work. I was not able to spend my daily time at the gym because I needed the nap. Food was often quick if not nutritious. A stakeout was proposed. The FBI would wait for Ellen outside the office. Jim from Physical Security would be on hand. It was late. The parking lot was dark, just dimly lit by the parking lot lights. A large planter cast a shadow across the sidewalk adding to the darkness. Ellen was working late. She was finally on her way out of the door. She was stopped and searched. Ellen had with her documents that were labled proprietary information. We now had enough for a search warrent.

A warrent was issued for Ellen's home and computers belonging to her as well as the family. Copies of the disk drives were made and Lori performed the forensic analysis of those. It was determined that digital copies of the data were made to USB thumb-drives and by burning to CD/DVD ROM. Another search of the house was made. There was no evidence of digital media in the home. Were they sent on ahead? Were they even real? So many questions were left unanswered. Lori found more company documents on the private laptop. These documents were marked proprietary information. Now we found evidence in the home and on personally owned devices.

An arrent warrent was requested. Ellen was already on the way to the airport with her husband and child. We knew her flight information. She was leaving the country tonight. We did not have the warrent in hand. We had to wait on a judge to sign off. The FBI was at the airport monitoring the situation. She passed through security check and was on her way to the terminal gate. Would we get there in time. The FBI approached Ellen at the termal gate. Trying to convince her not to board the plane. She knew they couldn't touch her without a warrent. She refused their pleadings. The plane was boarding. The FBI was stalling. Ellen rebuffed them and boarded the plane. The warrent was too late. She was on the plane and it was about to take off to lands far away. We missed our opportunity. My heart sunk. Was all this hard work, 80 hour weeks, for the last couple of months for nothing? Is there any hope of catching her on a connecting flight? Did we lose?

Time passed. I did get my forensic analsys certification from the SANS Institue. I did not have any more interesting cases such as this. It was quite boring really. I left the information security team to help the R&D department with policy implementation full time. I was working with Joe again. Still, not really friends but good acquaintances. Joe asked about the case. I let him down gentley. It was during my time working with the R&D team when I was laid-off. One of the departments was not selling product and the company chose to include IT staff with the lay-off. I left the company. I got married. Eight years after the investigation I received a letter for the Depart of Justice.

I got a call from the local Dallas FBI office. Ellen was stopped at the border trying to enter North Korea. Apparently she was red-flagged for travel. Ellen was taken into custody and sent to the USA to stand trial. Over the course of several months I got letters from the Department of Justice, black SUVs parked outside my home and finally one agent asked if I would testify. I agreed. I was told that I would be served with a subpoena to make it all official. The FBI agent wanted to serve it to me at work. I told him that we needed to meet outside of work and picked a restaurant close by. I informed the agents that I worked in the security operations center (SOC) and getting in is a pain. This was just easier. We had a nice lunch and I took home my subpoena. Over the next few weeks I spent many hours working with prosecutors, agents, etc. I often wondered what my new wife thought of all this activity. What did my neighbors think with all the black SUVs outside my house nearly every night. What did the postman think will all the corospondence from the Deparment of Justice? The trial began. I took the light-rail downtown to the federal courthouse. The trial lasted but five days, begining to verdict. My former directory and Lori gave testomy the first morning. I was called in after lunch that first day and told to return on day two. In total, I gave nearly a day-and-a-half testomy. There were three days of deliberation. The verdict came back, "not guilty". News articles reported the inablility for procescution to prove intent. The entire case was built on intent. Ellen claimed no such intent. I have no reason to doubt her assertion. I believe that God ensured justice was complete. I believe this verdict was in answer to my prayers at the begining of the trial -- that she not be punish if she was not guilty of selling secrets to our competators.

Don't think in here (9 Jan 2023)

My early career was working for a major semi-conductor company. I worked for many different groups while there. I supported manufacturing, test, engineering design, corporate IT, and information security. My career there spanned nearly 25 years if I count my earliest years as a contractor. Supporting manufacturing I learned that cost savings is king and many times management cannot be convinced with logic. This is one of those cases but I prevailed with a no-cost solution to a problem raised to me.

A manufacturing floor engineer called early one morning. He was breathless and I could tell that he was exasperated by something. The night before his test system failed and no one notified him. He was now a day behind and frustrated by the lack of action. I was new to the group and Larry knew that. Larry was normally a mild-mannered gentleman. I am sure his traditional Chinese upbringing had a lot to do with that. "My system failed again last night. Isn't there someway we can have a high availability system on a brand-new manufacturing floor?" Larry continued, "Nobody seems to want to help me. Can you find a solution?" In fact, I could. I told him of our standard solution. It was design specifically for situations like these. It cost $5,000 per node. Larry would take that offer to his management.

A little about the system Larry was managing. The test equipment was supported by two Sun Solaris workstations. They both connected to a common network attached storage system and could read and write data. One system was the primary, controlling the test equipment. The other system was running idle in case of failure. What we refer to as a warm spare. The test equipment would only talk to the Sun workstation with the primary IP address. If the primary failed, the engineer requested an IP address swap from our networking group. During normal business hours this could be a quick swap. After hours, though, required an on-call network engineer to be notified by a ticketing system, make a VPN connecting into the office and submit the changes requested. This could take an hour or more. We were told that an offline piece of equipment cost the company $100,000 an hour in lost productivity. You can see why manufacturing engineers took this seriously.

Larry's management came back. They did not want to spend the money on the commercial high-available software. It is not an expenditure; it is a cost savings. I asked, "Larry, how many times per year do you have a system failure?" His answer was a surprise low number, about four. If we use our number of $100,000/hr of lost productivity, then four times a year was $1 billion in savings if we assume just one hour of down time per incident. Larry took the numbers to his management. They were adamant; they did not want to pay for high-availability software. Larry inhaled sharply and blew it out all at once. "Isn't there something that anyone can do?" I asked Larry to let me think on it for a bit. I would get back to him with an answer.

From an early age I looked at life differently. I credit my interest in photography for some of that. There is more than one side to any subject and I had to find the other side of this one. Upon confirming the version of the operating system on his test equipment with Larry, I set upon a solution. I did not know if it would work but it had to. I built a test environment of Linux operating systems. Yes, I know the test equipment is using Sun Solaris. The Linux workstation had similar capabilities and was a free solution for a test environment. I read up on the virtual interface capabilities of both operating systems. They were very similar with only minor syntactical changes. I developed my test environment and debuted it to Larry. He was impressed. Speaking more rapidly than I ever hear him, Larry wanted to know when he could put this into production. I asked him to give me a week. I needed to create an alerting mechanism so he would know when a failure occurred and not rely on the manufacturing floor personnel. This allowed him to check up on the system to make sure the fail-over took place without incident.

The system was elegant in its solution. I started with the desired results. We wanted a system that could switch from the primary workstation to the warm spare system upon failure. We wanted this to occur automatically. We did want to rely on a person, be it manufacturing personnel or networking engineer to perform any task. The solution must be self-contained. Next we listed what we knew about the system. The test equipment was controlled by two Sun Solaris workstations. The test equipment could only communicate with one system at a time. The test equipment could only communicate by IP address; it did not have Dynamic Name Server (DNS) lookup server capabilities. Finally, we could readily configure the test equipment. My solution was to use three IP addresses: one, on which the test equipment would communicate, and two for the main interfaces of the controller workstations. I used simple scripts to test the communication between the two controllers. The secondary controller tests the network connectivity of the primary every minute. If the primary fails to respond, the secondary starts the process of configuring a virtual interface with the communication IP address. The test equipment should not ever know the initial-primary controller ever failed and just start communicated with the new controller. An email is then sent to Larry to let him know of the failure so that he may verify the change over. On startup, each system checks to see if the communication address is in use. If it is not yet in use, the first controller to boot configures the virtual interface and become the primary. It is not perfect but it is elegant in its solution

I performed the entire task without setting foot on the manufacturing floor. I never laid physical hands on the controller or test equipment. The entire process for me took place in my cubicle. One morning Larry was at our location for a meeting. He stopped by to introduce himself. When he entered my cubicle, he chuckled. I turned to ask what was so funny. He pointed to an open box I kept on my shelf. Within that box was a piece of paper upon which I printed, "Don't think in here!" Now that box had multiple meaning to me. The one most relevant to this occasion was don't settle for the standard answer. Look for innovative solutions. You just might be surprised.

Failure is not an option (16 Jan 2023)

It was the early 2000s and it was not since I joined the company's Information Security Team. I already gained a reputation at the corporate IT level for getting things done. I did not play politics and truely only wanted what was best for the company. Many of my coworkers had their own agendas. That is fine, but all request for production had to go through me for InfoSec approval; and I asked a lot of questions.

My director asked me to take over a project from one of my fellow InfoSec team mates. This is not an unusual request, we traded projects often. It sometimes helps to get a fresh set of eyes on a project to help move it forward. This project was to replace the current Windows™ client firewall software. Steve introducted me to the client firewall team when I attended the first meeting. He informed them that I was taking over the project from an InfoSec standpoint. The client firewall team wanted to jump right into work but I put the breaks on. I needed to read the charter as given by the VP of IT Operations and review what took place to date. We would start anew the next week once I read the documents. Speaking quickly and abruptly on member questioned my motives. I replied using my best "nighttime FM DJ voice" that my motives were to do what was best for the company and based on the VP directive. I then asked wryly, what her's were. On that note I left the conference room.

The next week I walked in the conference room prepared having read all the plans to date. It was my opinon that if we just upgraded the existing software in place that we could achieve the desired outcome. It was pointed out by Randy, the project IT leader that was not an option. His voice started strong and then became quiet. Given Randy's tone I guessed that he agreed with me. Our VP's written instructions were to replace the existing software. It was too bad. The updated software was much more capable than the eventual replacement. It performed exactly as desired and was centrally managed. That last part was the issue with the version that we were currently running. The old, unsupported version was not centrally managable. We deliberated as a team. Waying the advantages and disadvantages to each solution. I could tell from the tone of the room that we all understood the best solution was not on the table but we could only pick from what was available to us. It only took us a matter of weeks to pick a candidate for evaluation. Randy from the IT Operations team built the managment server and we deployed the client to the client firewall team's computers. It was decided that the interface was not great but the solution met most of the requirements set about by the team. It was agreed by all that Randy would present our findings to our VP.

I was broadsided by Steve when I got back the office. He was speaking agressively, and rather loudly. How could I have allowed that product to be approve? At this point, I knew the bottle-neck was Steve and not any issue with the rest of the team. I tried to explain as calmly as I could that this was the best solution of our given options. Steve was adiment about upgrading the existing software and was not going to take no for an answer. I simply informed him that was not an option given the directive from our VP. Steve had some unkind words about our VP and the directive. While I agree with his sentiment, I had to move us forward in the project and not joust at windmills.It was about this time that my director, Brian, walked in. I had the cubile next to Brian's highwall office. Inquisitevely Brian asked what was going on. I allowed Steve to state his possion. The words rushed out of his mouth like water cascading over a waterfall. His voice still elevated like he just walked out of a rock concert and was still struggling to hear his own voice. Brian gently close the door so we could talk. Steve laid out his objections point by point; filled with emotion. I get it, Steve was not pleased with the decision. Brian then asked for my point of view. It was simple. Our VP clearly stated to replace our current client firewall. Upgrading it in place was not an option. We reviewed the remaining options and chose the one that most closely aligned with InfoSec goals and was centrally managable. Brian quipped that it seemed like a done deal. Steve stormed out of the office. I think he felt betrayed. I had a job to do, I simply did my job.

From the onset of taking over this project, it was my goal to make sure that we were successful. I told Brian when he asked me to take it on that failure was not an option. This was the attitude that I started my career with when I left home after college. My parent told me as I was walking out the door, headed for the big city, that the door was always open if things didn't work out. I simply stated "failure is not an option." I heard my Dad snicker under his breath. Having just finished his book I understand now why he thought that was funny. I, afterall, was my father's son.

I struggled more than most for the same results(30 Jan 2023)

I never liked school. From a young age we moved quite frequently. This made it difficult for me to establish long term relationships. Looking back on it, I gave up on that a long time ago. I was satisfied with new short term friends that I would abandon with our next move. It was just a fact of life for me. I also had undiagnosed ADHD and Dyslexia. At least I was never told I had these problems. I just struggled through. I thought everyone else was the same. Finally, I was always a small boy and often picked on. The combination just made school an uncomfortable experience for me.

In elementary school I taught myself to deal with the distractions of the classroom. I had a technique that I developed to block out the sounds and movements of the other kids. In the third grade I was analyzed by a doctor at the school’s insistence. They checked my eyes, hearing, and various other things. The only thing that I took away from all the poking and prodding, I had a slight curvature in my spine. Great, do I have scoliosis, too? Nothing changed. I was allowed to return to class. For all of that, I was held back and not allowed to advance with my classmates. Did I tell you I did not like school?

I started my Middle School career at Estee Middle School in New York. I tell you, my first year was uneventful. I made no connection with anyone. I showed up, did my time and left. I was almost like a prison sentence. My second year, grade seven, was much better. This despite the fact that I knew my family was moving again. I had no particulars just that we were moving once my father’s transfer was approved. There was a group of boys which whom I started hanging around: Steven, Doug, Michael, and Bruce. Later Aaron joined the group as well. I loved my new friends; cherished them even. We had many of the same classes together, including Phys. Ed. and shop. Alas, the time came to move and we traveled from New York to rural Texas. I tell you, what a change. I was still picked on but I did make some new friends: Rex and Roy (the twins), Paul, Randy, William, just to name a few. These friends was mostly able to carry on to college.

High school was uneventful all throughout. My friends I made in middle school were still my friends, well except for William who moved away. I started working for a fast food chain in my 10th grade year. I did not realize it until much later but I was not picked on in high school I never did figure out what changed. I worked all the way up to graduation. Even after being held up at gunpoint. I struggled through to make grades but I did manage to make it all the way through school.

I was off to college. My older brother, Grant, asked me what I was studying. Computer Science, I replied. I quote: "You are going to study computer science with your ADHD!?!" Shocked I proclaimed that I did not have ADHD and my mom laughed. Not just a little. No one told me. I just assume everyone was like me. We were kids, we had excess energy. That was normal. Apparently, I was well above average. I was taking an accounting class taught by a local CPA subbing in because one of the professors was on leave. I did not do great but I got solid Bs and Cs. I was happy with that. My teacher approached me one day and asked if I was ever diagnosed. I had no idea what she meant. She explained to me the mistakes that I was making on the homework assignments and thought that I might be Dyslexic. No, I was never diagnosed; or tested for that matter. She made an appointment with the university resource office to have me tested. Great! Not only did they confirm my ADHD but now I have Dyslexia, too? Well, at least that explains the numerical transpositions. Reflecting on my educational life, it explains a hell-of-a-lot more. Well, at least I have an answer to my difficulties. Now that I was aware, I could compensate. Why does life have to be so hard. I was able to graduate college with a 3.8GPA. Not bad for someone with new found challenges.

I struggled more than most for the same results, well not even as good most of the time. I had trouble reading. That was a big problem for a university education. I had trouble with spelling. I love computers and autocorrect. I refused to allow my challenges hold me back. I stated in another blog that failure was not an option. This is a long standing mantra to myself. Adapt and overcome was another mantra. Always moving forward to achieve my goal. I graduated thought I never became a programmer. My field of choice was system administration. I did it well for many years. With hard work, determination, and the ability to learn, not just classroom but myself, I was able to succeed. I have no special talents. Anyone can do what I accomplished. I quite often make it my mission help those coming up behind me see their value, see their worth, and know that they can make it too.

Avoid having your ego so close to your position that when your position falls, your ego goes with it. -Colin Powell (7 Feb 2023)

All managers were away at an off site meeting. Meaning they were playing golf. If this were any average day then that would be fine. This did not turn out to be an average day. With no management on site and no senior analyst willing to make the bid decisions, I stepped up. Wrong or right, I did what I thought was best for the company and the data we steward.

On this day we had a power outage. We had multiple power feeds to the facility and we had battery backups to carry us over until the secondary feed kicked in. Today we had multiple failures. Our primary feed was interrupted. At first we did not think anything of it. We waited to switch to the secondary feed. After a bit, about five minutes I estimate, Carl checked on the data of the switch over. I do not know what drove him to do so. I am glad he did. I followed Carl to the front entrance of the data center. Carl went into the battery room. "Well, what does it look like?" I asked. We never switched over the secondary feed and Carl was unsuccessful at getting it to switch manually. We officially had a double failure.

Standing there in front of the data center, we discussed our options. Carl informed that the battery backup was rated at thirty-minutes. Great, now we have a starting point. The power feed was out about five minutes to this point. I remember turning to Chris, technically he was more senior that I, and asked what we should do. Chris took chart of trying to contact our management. I devised a backup plan as I suspected that management would not answer the phone on the golf course. I was right. On the assumption that we had thirty total minutes of battery time and that we used about five of those already, I decided that we would start to shutdown the data center in about fifteen minutes unless power was restored. Our clock ran out. Fifteen minutes was up. I instructed Chris to have the other start shutting down computers in the data center to preserve the data. He refused. I, however, executed my plan on those systems for which I was responsible. I was able to shutdown all but two of the file servers. Twenty-three of 25 ain't bad. I ran out of time. Without knowing it, we over loaded the capacity of the batteries to maintain the data center for a full thirty minutes and I ran out of power.

City power was eventually restored and the data center was returned to production. In the shutdown, I only lost two disk drives. With the RAID systems in place that means there was no data loss. I replaced the disks from stock and called our supplier for an RMA number. Management returned sometime after lunch. I was in my office working on the RMA and writing up my notes to present to Gale, my department manager. I was not among those who swarmed management in a flurry to tell them what happened. About three o'clock that afternoon Gale called me into his office. I asked for a second as I pressed the enter key to send him my report on the event as I saw it. "I am on my way."

When I got to Gale's office, I told him I sent him a report on the incident in an email. He took a moment to read and asked me a few questions. Who told me the batter was rated for thirty minutes? Who made the decision to finally shutdown the data center? When did we know that secondary feed did not switch over? What did I pick fifteen minutes? That last one I had to answer sheepishly. I felt that I needed ten minutes to shutdown all the file servers cleanly and I took a SWAG (some wild ass guess). Gale's response was that I did the right thing for the right reasons. He thanked me for taking charge. He asked why I did not wait to hear back from the management team. I simply informed him that I knew they were on the golf course. I play golf. I know that it is bad form to leave your ringer on when out on the course. That I felt the likelihood of contacting management in time for a meaningful response was near zero. Was I wrong? Feeling put on the spot, Gale simply answered, "no."

Those of use with ADHD are uniquely qualified to handle crisis situations. We are wired for it. There are studies that show we are able to take in a lot of information from many sources and quickly digest it. Our creative mind kicks in to hypothesize solutions, and once decisions are made, our ability to hyperfocus on the task well past the point where mere mortals started to fade. We sometimes are afraid to act. We fear rejection. We often suffer from uncontrollable anxiety. I was able to control those negative emotions, make a decision, and execute despite objections from my peers. I did what I thought was right. I think that I was able to do this because I was grounded in my own belief system. I could stand on that and say, this is why, right or wrong. My ego is not tied to my beliefs so if I was wrong, I am able to change in light of new facts.