My Two Census

Formerly the non-partisan watchdog of the 2010 US Census, and currently an opinion blog that covers all things political, media, foreign policy, globalization, and culture…but sometimes returning to its census/demographics roots.

Posts Tagged ‘HQ’

Anatomy of a Paper-Based Operations Control System (PBOCS) failure…

Wednesday, April 21st, 2010

Below are e-mails obtained by MyTwoCensus.com sent from Brian Monaghan and Barbara M. Lopresti at Census Bureau Headquarters to every regional Census Bureau office in America that describe IT systems failures:

From:
Brian Monaghan/FLD/HQ/BOC

To:

FLD Regional Directors

Cc:

FLD Deputy Regional Directors List, Barbara M LoPresti/TMO/HQ/BOC@BOC, Chad G Nelson/TMO/HQ/BOC@BOC, Janet R Cummings/FLD/HQ/BOC@BOC, Gail A Leithauser/FLD/HQ/BOC@BOC, Marilia A Matos/DIR/HQ/BOC@BOC, Annetta Clark Smith/DMD/HQ/BOC@BOC, Michael T Thieme/DMD/HQ/BOC@BOC, Timothy P Olson/FLD/HQ/BOC@BOC

Date:

04/20/2010 09:52 AM

Subject:

PBOCS and NRFU

OK, folks…   here’s where we are as of 8:45 a.m. Tuesday morning  -

As of 6:00 p.m. or so Monday evening, the last of the LMR automated removal occurred.  On Friday, April 23, there will be a PBOCS deployment which will include the reports of LMRs since Monday.  Those reports will then be available for clerical line-through of LMRs on the assignment registers (which are hopefully being printed by then).

We are expecting all of the even numbered AAs to have their reports (listings, labels, etc.) generated in the system by 11:00 a.m. this morning.

The system continues to be somewhat unstable, so at midnight tonight we need all LCOs and RCCs to get off PBOCS and stay off until Thursday morning (we hope).  That will give us a clean opportunity to generate the majority of reports for the odd numbered AA’s (we hope).  So…  no users on the system starting at midnight tonight and lasting through Wednesday.

Our # 1 priority is to get all of the reports generated and copied to an alternative printing site, so that if PBOCS goes down, the LCOs will still be able to print materials needed for NRFU assignment prep.   Once the even numbered AAs have all of their reports generated (by 11:00 a.m. this morning), we will begin the process of exporting the files to an alternative print site.   Several additional meetings need to occur to work through all of the details, but our hope is that DOTS will be testing this alternative printing site in, say, one LCO per region…  ideally nearby the RCC so your LSC can observe…  either this afternoon or tomorrow morning.  It’s not clear at this point whether we will be able to pull everything together that quickly.

Bottom line is that we are still planning for the LCOs to begin printing assignments for the even numbered AAs Thursday morning…  either through PBOCS or the alternative print site.  At this time, we are assuming all other PBOCS users will also regain access to the system Thursday morning.   We have asked that odd numbered AAs be made available on a flow basis of some sort…  groups of LCOs or regions…  rather than waiting until all reports are generated to make them available for printing. This weekend will be a huge crunch time for the LCOs…  all hands need to be on deck…  as they prepare assignments for all of the even numbered AAs and as many of the odd as possible.

Please make sure the LCOs are firing on all cylinders with NRFU map printing. That task is outside of PBOCS, so the downtime tomorrow will not be a problem.  It’s really critical to get this job done ASAP, so that the printers in the LCO are not tied up with NRFU maps, and are available for assignment prep.    If you cannot get all NRFU maps done by COB Wednesday, give top priority to the even numbered AAs, so assignment prep can be completed for work headed to the field first thing next week.  An added impetus to the NRFU map printing work is that there is a remote chance that LCOs may be able to start assignment prep for even numbered AAs tomorrow (Wednesday) if we are able to get the alternative print site set up, files exported, systems tested in some LCOs, and instructions prepared.   LCOs which have completed NRFU map printing will be likely candidates for this somewhat unlikely event.

We can talk more at the RD Conference Call this afternoon, or call me if you have an immediate concern.

From:
Brian Monaghan/FLD/HQ/BOC

To:

FLD Regional Directors

Cc:

FLD Deputy Regional Directors List, FLD Decennial Branch Chiefs, FLD Decennial Assistant Division Chiefs List, Marilia A Matos/DIR/HQ/BOC@BOC, Barbara M LoPresti/TMO/HQ/BOC@BOC, Michael T Thieme/DMD/HQ/BOC@BOC, Chad G Nelson/TMO/HQ/BOC@BOC

Date:

04/14/2010 09:48 AM

Subject:

PBOCS

As you may know, PBOCS went down last night.  The 40 LCOs that were scheduled to be ingested did not get ingested.  PBOCS is back up this morning and available for your use, but the concerns about instability remain.

We must do the following to prepare for NRFU:

PBOCS will be taken down tonight at 8:00 p.m., and will not be available again until Monday morning, April 19. Hopefully, minimizing the number of users and uses will increase the stability of the system, allowing the full ingest of all LCOs to be completed over the next several days.  As you heard at the Regional Directors’ Conference, this is a critical first step in the process of preparing for NRFU assignment prep.

DOTS will be sending out a separate notice to you and your automation folks, and each of the Decennial Branch Chiefs will issue ops logs with suggestions and cautions about getting through the next several days.  For example,  it’s critically important not to send completed work to the processing office unless it has been checked out through PBOCS.  If you box up and send in ICRs/MCRs without going through the formal PBOCS check-out process, we will lose the critical linkage with their Group Quarters.   We will be asking you to hold completed work in the office until PBOCS is back up and running.  Of course, work on all operations can and should continue in the field.

This will be a really important time for the LCOs to stay as organized and systematic as possible…   labeling and sorting piles of completed and pending work in  a way such that, when PBOCS is made available, we can rapidly recover.  If work needs to go to the field while PBOCS is down, the LCOs will need to manually track the assignments, so they know who has what, and when  they got it.  Once PBOCS is made available on Monday, the LCOs will need to key in this information to get the system caught up.

Thanks for your patience as we work through these challenges.

From:
Brian Monaghan/FLD/HQ/BOC

To:

fld.regional.directors@census.gov

Cc:

fld.decennial.assistant.division.chiefs.list@census.gov” <fld.decennial.assistant.division.chiefs.list@census.gov>, “Barbara M LoPresti” <barbara.m.lopresti@census.gov>, “Chad Nelson” <chad.g.nelson@census.gov>, “fld.deputy.regional.directors.list@census.gov” <fld.deputy.regional.directors.list@census.gov>

Date:

04/08/2010 05:57 PM

Subject:

Fw: PBOCS System Outage starting Friday April 9th at 500pm ET.

We need to shut down PBOCS at 5:00 p.m. Friday, April 9, instead of waiting until midnight.  We had a lengthy discussion today and, as you can imagine, time is a critical commodity.  Lots of work has to be done in preparation for NRFU, and if it means an extension for ETL or delays in check-in of GQE and UE, so be it.

Call me if you have any questions or just need to vent.  We wouldn’t be doing this full weekend shutdown if it wasn’t really necessary.

    Inactive hide details for Barbara M LoPresti Barbara M LoPresti

—– Original Message —–
From: Barbara M LoPresti
Sent: 04/08/2010 05:02 PM EDT
To: Brian Monaghan; Chad Nelson; Janet Cummings; Gail Leithauser; Annetta Smith; Michael Thieme; Pamela Mosley; Marilia Matos; Arnold Jackson
Cc: Thomas McNeal; Curtis Broadway
Subject: PBOCS System Outage starting Friday April 9th at 500pm ET.
Brian,
In the 430 meeting today, Tom and Curtis felt it was best to take the Pbocs system down at 5::00 pm eastern time on Friday, April 9th (tomorrow).
Please let me and Chad know when you have informed the RDs and then we will get a DOTS message out to the RCCs.
Thanks
Barbara


The Suitland Files: Inside The Census Bureau (Part 1)

Monday, November 2nd, 2009

This post is dedicated to the memory Dr. Hunter S. Thompson. The other night, I caught Alex Gibney’s biographical documentary Gonzo: The Life and Work of Hunter S. Thompson, and I was once again reminded how exciting political journalism can be, especially when it’s written in the first person. So, here is the first of two installments detailing my trip to the U.S. Census Bureau Headquarters in Suitland, Maryland, written in a style that I hope would make Dr. Thompson smile:

President Obama must have sensed that I had a whole lot of questions for him, because just as soon as I arrived for my tour of the White House, Marine One arrived to whisk him off to Andrews Air Force Base for a trip to Boston. Nonetheless, after five months in other parts of the world, it was good to be back in the District, which was just as vibrant as when I left it in May.

After a post-White House pit stop at Potbelly Sandwich Works, I jumped on the red line at Metro Center for a minute before changing at Chinatown to the green line that would coast me all the way out to Suitland, Maryland, home of the U.S. Census Bureau’s  HQ. It’s a long and lonely ride out to Suitland, as it’s just about as far as the Metro can take you from any of DC’s attractions. Once the Metro stopped at Anacostia I couldn’t help but be wary, knowing that I was in the highest-crime district in an already high-crime city.

When I exited the Metro at Suitland, I noticed that my fellow riders (no less than three individuals walking with painful limps, a pair of girls who looked to be no more than seventeen –each with a baby in their arms, and a blind man who lacked a cane and got from point A to point B by only by sticking his hands out to guide him –which seems to be a death wish in the vicinity of active train tracks) all seemed depressed. And watching them made me depressed, so I scrambled onward. I walked through an endless parking garage, the whole time during which I was convinced that Deep Throat would sneak up on me from behind. When I finally made my way to its entrance, there she stood, looking completely out of place, like a princess at a soup kitchen, the glass-paneled behemoth that was completed in 2006 and holds unquantifiable amounts of data. When Census employees click their heels together three times and say “There’s no place like home,” this is where they land.

I’d been told to arrive early for my meeting, because after all, I was meeting with some very high-level bureaucrats, and they had, you know, things to do. But I still had forty minutes, which was too early, so I figured I’d take a lap around the building to kill some time. Just as I walked to the edge of the building’s iron-gated perimeter, I peered through the main entrance and saw a familiar face, or should I say a familiar pair of spectacles and a familiar gray mustache. “No way, it can’t be,” I thought to myself. But it was. I knew it in an instant. My heart started to pound. I could feel the sweat dripping down my neck. I wanted to loosen my tie but I suddenly was no longer in control of my hands, which were now involuntarily shaking. Of the thousands upon thousands of employees of the U.S. Census Bureau who are based at headquarters in Suitland, here I was, standing beside the top dog, the king of the castle, the questionnaire czar, the big kahuna, el estadistico grande, the Don Juan of Censusland…it was none other than Dr. Robert M. Groves himself.

Still in a trance, I strode right past the security guards (who were obviously doing a great job keeping the place safe and secure) and shouted “Dr. Groves!” with the enthusiasm of a kid who was about to get his baseball signed by Babe Ruth. (I mean, Census bloggers need heroes too.) Groves stopped dead in his tracks and stared me down. He had the look of a man who’d just been caught by TMZ with his pants down, but it was really just the inquisitive ambivalence of responding to someone who shouts your name as if you’re old friends when in reality you’re hardly even acquaintances.

Surely he didn’t recognize me, as I was sporting a mustache and glasses myself, two accessories I lacked during our only other encounter, which was at his confirmation hearing back in May (The ‘stache and specs were just a coincidence, not an elaborate homage, I swear!). During my first brush with Census royalty, Groves, all but assured of his Senate confirmation said to me, “You should come by Suitland some time soon and I’ll give you a personal tour.” I told him right then and there that I would take him up on his offer and hold him to his word.

As i was still crippled with fear and verbally paralyzed, Groves said to me, “Ah, I wanted to come to your meeting, but I was called to go somewhere at the last second.” You would think that I would be utterly dejected by this, but this wasn’t the case at all, as I had no idea that Dr. Groves was even considering a meeting with me, so this was much more than I’d bargained for. Still overjoyed, all I could think was,  ”Damnit, why didn’t I bring my camera?”

Not knowing what to do as the power of speech suddenly returned to my body, I asked him for his business card. He fumbled around his wallet for a few seconds and told me he was out. Ostensibly he doesn’t want to be on my speed dial, so he played it safe with a solid excuse. That, I can totally understand. (I wouldn’t want me on speed dial either.) Clearly in a rush (his driver was waiting for him), Groves parted with me by saying, “You’re doing a good job.” As I blurted out a terse “thank you, ” he was already on his way.

Still in a relative daze, I only floated back to the real world only when I felt the heavy hand of a security guard on my shoulder. Even if the rest of my Census Bureau HQ experience went to shit, at least I had the approval of the one person who mattered most in Suitland’s Glassy-Glowing-God-like Monolith.

Fedex-gate pt. 3: FedEx substitues for MS Word

Thursday, June 11th, 2009
Guide for Document Reviewers 5-14-09.pdf (page 1 of 2)MyTwoCensus obtained an internal memo from the Census Bureau that directs employees to use FedEx every time they want to make a change to a document

Hmmm, in most worlds using Microsoft Word’s “track changes” feature would suffice, but apparently not at the Census Bureau’s Suitland, Maryland headquarters.