My Two Census

Formerly the non-partisan watchdog of the 2010 US Census, and currently an opinion blog that covers all things political, media, foreign policy, globalization, and culture…but sometimes returning to its census/demographics roots.

Anatomy of a Paper-Based Operations Control System (PBOCS) failure…

Below are e-mails obtained by MyTwoCensus.com sent from Brian Monaghan and Barbara M. Lopresti at Census Bureau Headquarters to every regional Census Bureau office in America that describe IT systems failures:

From:
Brian Monaghan/FLD/HQ/BOC

To:

FLD Regional Directors

Cc:

FLD Deputy Regional Directors List, Barbara M LoPresti/TMO/HQ/BOC@BOC, Chad G Nelson/TMO/HQ/BOC@BOC, Janet R Cummings/FLD/HQ/BOC@BOC, Gail A Leithauser/FLD/HQ/BOC@BOC, Marilia A Matos/DIR/HQ/BOC@BOC, Annetta Clark Smith/DMD/HQ/BOC@BOC, Michael T Thieme/DMD/HQ/BOC@BOC, Timothy P Olson/FLD/HQ/BOC@BOC

Date:

04/20/2010 09:52 AM

Subject:

PBOCS and NRFU

OK, folks…   here’s where we are as of 8:45 a.m. Tuesday morning  -

As of 6:00 p.m. or so Monday evening, the last of the LMR automated removal occurred.  On Friday, April 23, there will be a PBOCS deployment which will include the reports of LMRs since Monday.  Those reports will then be available for clerical line-through of LMRs on the assignment registers (which are hopefully being printed by then).

We are expecting all of the even numbered AAs to have their reports (listings, labels, etc.) generated in the system by 11:00 a.m. this morning.

The system continues to be somewhat unstable, so at midnight tonight we need all LCOs and RCCs to get off PBOCS and stay off until Thursday morning (we hope).  That will give us a clean opportunity to generate the majority of reports for the odd numbered AA’s (we hope).  So…  no users on the system starting at midnight tonight and lasting through Wednesday.

Our # 1 priority is to get all of the reports generated and copied to an alternative printing site, so that if PBOCS goes down, the LCOs will still be able to print materials needed for NRFU assignment prep.   Once the even numbered AAs have all of their reports generated (by 11:00 a.m. this morning), we will begin the process of exporting the files to an alternative print site.   Several additional meetings need to occur to work through all of the details, but our hope is that DOTS will be testing this alternative printing site in, say, one LCO per region…  ideally nearby the RCC so your LSC can observe…  either this afternoon or tomorrow morning.  It’s not clear at this point whether we will be able to pull everything together that quickly.

Bottom line is that we are still planning for the LCOs to begin printing assignments for the even numbered AAs Thursday morning…  either through PBOCS or the alternative print site.  At this time, we are assuming all other PBOCS users will also regain access to the system Thursday morning.   We have asked that odd numbered AAs be made available on a flow basis of some sort…  groups of LCOs or regions…  rather than waiting until all reports are generated to make them available for printing. This weekend will be a huge crunch time for the LCOs…  all hands need to be on deck…  as they prepare assignments for all of the even numbered AAs and as many of the odd as possible.

Please make sure the LCOs are firing on all cylinders with NRFU map printing. That task is outside of PBOCS, so the downtime tomorrow will not be a problem.  It’s really critical to get this job done ASAP, so that the printers in the LCO are not tied up with NRFU maps, and are available for assignment prep.    If you cannot get all NRFU maps done by COB Wednesday, give top priority to the even numbered AAs, so assignment prep can be completed for work headed to the field first thing next week.  An added impetus to the NRFU map printing work is that there is a remote chance that LCOs may be able to start assignment prep for even numbered AAs tomorrow (Wednesday) if we are able to get the alternative print site set up, files exported, systems tested in some LCOs, and instructions prepared.   LCOs which have completed NRFU map printing will be likely candidates for this somewhat unlikely event.

We can talk more at the RD Conference Call this afternoon, or call me if you have an immediate concern.

From:
Brian Monaghan/FLD/HQ/BOC

To:

FLD Regional Directors

Cc:

FLD Deputy Regional Directors List, FLD Decennial Branch Chiefs, FLD Decennial Assistant Division Chiefs List, Marilia A Matos/DIR/HQ/BOC@BOC, Barbara M LoPresti/TMO/HQ/BOC@BOC, Michael T Thieme/DMD/HQ/BOC@BOC, Chad G Nelson/TMO/HQ/BOC@BOC

Date:

04/14/2010 09:48 AM

Subject:

PBOCS

As you may know, PBOCS went down last night.  The 40 LCOs that were scheduled to be ingested did not get ingested.  PBOCS is back up this morning and available for your use, but the concerns about instability remain.

We must do the following to prepare for NRFU:

PBOCS will be taken down tonight at 8:00 p.m., and will not be available again until Monday morning, April 19. Hopefully, minimizing the number of users and uses will increase the stability of the system, allowing the full ingest of all LCOs to be completed over the next several days.  As you heard at the Regional Directors’ Conference, this is a critical first step in the process of preparing for NRFU assignment prep.

DOTS will be sending out a separate notice to you and your automation folks, and each of the Decennial Branch Chiefs will issue ops logs with suggestions and cautions about getting through the next several days.  For example,  it’s critically important not to send completed work to the processing office unless it has been checked out through PBOCS.  If you box up and send in ICRs/MCRs without going through the formal PBOCS check-out process, we will lose the critical linkage with their Group Quarters.   We will be asking you to hold completed work in the office until PBOCS is back up and running.  Of course, work on all operations can and should continue in the field.

This will be a really important time for the LCOs to stay as organized and systematic as possible…   labeling and sorting piles of completed and pending work in  a way such that, when PBOCS is made available, we can rapidly recover.  If work needs to go to the field while PBOCS is down, the LCOs will need to manually track the assignments, so they know who has what, and when  they got it.  Once PBOCS is made available on Monday, the LCOs will need to key in this information to get the system caught up.

Thanks for your patience as we work through these challenges.

From:
Brian Monaghan/FLD/HQ/BOC

To:

fld.regional.directors@census.gov

Cc:

fld.decennial.assistant.division.chiefs.list@census.gov” <fld.decennial.assistant.division.chiefs.list@census.gov>, “Barbara M LoPresti” <barbara.m.lopresti@census.gov>, “Chad Nelson” <chad.g.nelson@census.gov>, “fld.deputy.regional.directors.list@census.gov” <fld.deputy.regional.directors.list@census.gov>

Date:

04/08/2010 05:57 PM

Subject:

Fw: PBOCS System Outage starting Friday April 9th at 500pm ET.

We need to shut down PBOCS at 5:00 p.m. Friday, April 9, instead of waiting until midnight.  We had a lengthy discussion today and, as you can imagine, time is a critical commodity.  Lots of work has to be done in preparation for NRFU, and if it means an extension for ETL or delays in check-in of GQE and UE, so be it.

Call me if you have any questions or just need to vent.  We wouldn’t be doing this full weekend shutdown if it wasn’t really necessary.

    Inactive hide details for Barbara M LoPresti Barbara M LoPresti

—– Original Message —–
From: Barbara M LoPresti
Sent: 04/08/2010 05:02 PM EDT
To: Brian Monaghan; Chad Nelson; Janet Cummings; Gail Leithauser; Annetta Smith; Michael Thieme; Pamela Mosley; Marilia Matos; Arnold Jackson
Cc: Thomas McNeal; Curtis Broadway
Subject: PBOCS System Outage starting Friday April 9th at 500pm ET.
Brian,
In the 430 meeting today, Tom and Curtis felt it was best to take the Pbocs system down at 5::00 pm eastern time on Friday, April 9th (tomorrow).
Please let me and Chad know when you have informed the RDs and then we will get a DOTS message out to the RCCs.
Thanks
Barbara


Tags: , , , , , , , , , , , , , ,

13 Responses to “Anatomy of a Paper-Based Operations Control System (PBOCS) failure…”

  1. TR Says:

    Deja vu… sounds very similar to the DAILY system outages during Address Canvassing.

  2. CL newbie Says:

    Just got through the blitz CL training. I have no supervisory experience. I now have 5 days to actually learn everything from the regulations/manuals, to the point where I will even feel comfortable advising and training my enums. Everyone in my CL class passed. Everyone. Some of them had trouble reading through the 6th grade level diction in the material. Some of them were texting the entire time. Some of them slept. Most of them didn’t know what “business casual” means. The final CL “test” wasn’t even graded until AFTER we met our FOS’s and got our CLDs assigned. The girl sitting next to me got a 4/20… ON A 40 MINUTE, 20 QUESTION, OPEN BOOK TEST! The FOS who taught… i mean “verbatim lectured” the class was not a dumb man, so it’s obvious he was told by the higher ups to go easy on releasing people from the training. This of course got my neck hairs up… and so thank you for this site. Moving forward I will be considering how to mash 40hrs into every week while going back on the job hunt… and doing the best by my crew… who will probably be expecting 12 weeks of work as I was.

  3. Stephen Robert Morse Says:

    To everyone reading this: There are obviously massive problems throughout the country. Feel free to send over anything you know about. Your anonymity will be protected on this site. SRM

  4. HermHollerith Says:

    Wish List:
    A webcam in Brian Monaghan’s office,
    streaming audio of all his phone calls and
    live internet publication of all his incoming and outgoing email.

  5. HermHollerith Says:

    The most recent email from Monaghan refers to “clerical line-through of LMRs on the assignment registers”.
    This describes the absence of a computer system,
    not a problem with a poorly developed and tested computer system.

    Assignment registers totaling about 50 million addresses are supposed to be printed. Lists of Late Mail Returns
    (LMRs), perhaps around 2 million, are also supposed to be printed. Clerks in the Local Census Offices (LCOs)
    are supposed to manually compare the two lists. Clerks are to use a pen to strike through addresses on the longer assignment register lists that have mailed back a Late Mail Return. Very 20th century. Maybe not too bad for the 1980 Census but truly awful for the larger 2010 population. Also possibly susceptible to political skullduggery at the local level. If the clerk is texting or watching video on a mobile device, there will be even more mistakes.

  6. Techy Says:

    Herm: The lining through has been in place for several months. This is due to several factors. The two big ones being that the printing of the LHRs began YESTERDAY (and at least here, went VERY smoothly), and that the Census is STILL receiving questionnaire forms back from people.

    Now, just yesterday we printed a good 40,000 pages worth of information for the NRFU operations. When we get new information into the system on addresses that we don’t have to see but are on the registers we can do one of three things:

    1) Reprint the pages with the registers, thus potentially wasting thousands of man hours and thousands of pages of paper across the country.
    2) Ignore the new data and not change or mark anything, and have enumerators go visit those addresses anyway, wasting again, thousands of dollars of manhours and thousands of pages of paper on questionnaires, in ADDITION to making people who DID respond ANGRY because we’re interviewing them on something they already did.
    3) Have a clerk take two seconds to line through a register, not wasting thousands of manhours and pages of paper, but deeply lowering the number of #2 pencils throughout the country.

    As you can see, 3, which is lining out the addresses manually, is the best option.

  7. Techy Says:

    As for texting or watching a mobile video while working… that doesn’t happen. At least not here. And any office that allows that while working isn’t a good one, in ANY professional setting. I personally would fire anyone doing such a thing without a VERY good reason, no matter what job I was at, including this one.

  8. HermHollerith Says:

    Techy,

    How can it take only two seconds to line through a register? Doesn’t the clerk need to assemble the LMR list,
    the assignment register and the #2 pencil, then compare each address on the LMR list to the assignment register? Maybe it takes only two seconds to strike through the address on the assignment register but surely it takes longer to find the address on the assignment register? And this must be done about 2 million times around the country.

    Yes, reprinting the registers would waste paper. But you are wrong when you say thousands of pages of paper;actually many millions of pages would be wasted.

    None of this clerical nonsense would be necessary if the HHCs worked.
    And the project would be relatively paperless.

  9. Techy Says:

    “How can it take only two seconds to line through a register? Doesn’t the clerk need to assemble the LMR list,
    the assignment register and the #2 pencil, then compare each address on the LMR list to the assignment register? Maybe it takes only two seconds to strike through the address on the assignment register but surely it takes longer to find the address on the assignment register? And this must be done about 2 million times around the country.”

    The actual lining through takes two seconds. The prep work, as we did it today… takes about 3 minutes. Compared to the 4 hours it would take to reprint every address list MULTIPLE times, PLUS the hours it would take to remove the old lists and insert new ones, PLUS the cost in paper and toner which is less when just crossing out over reprinting multiple times.

    In short: Reprinting the address lists multiple times is stupid. Printing or referencing an LHR report and lining through the exisiting addresses in a register with a pencil is smart, quicker, and cheaper. Period.

    “None of this clerical nonsense would be necessary if the HHCs worked.
    And the project would be relatively paperless.”

    This I agree with

  10. My Two Census » Blog Archive » FOIA Request: Give us the e-mails of the following people who should be held responsible for tech failures Says:

    [...] MyTwoCensus.com is hoping to get to the bottom of the Census Bureau’s IT woes: [...]

  11. LCO-AM Says:

    We’re operating 24/7 trying to prepare NRFU binders.

    Does anyone know if the last census was this poorly managed and organized?

    My heart goes out to all the people working at the local census offices.

    I’m there with you………….

  12. Trying2MakeSenseof2010Census Says:

    Hey,

    Thanks for the sympathy. I was called 3 weeks ago and hired as a renumerator. I took the job thinking it would be a nice way to meet people, learn my new neighborhood and neighbors and of course make a decent salary for a temporary part-time job – wow was I wrong.

    The person who is our crew leader cannot say words like enumerator, two thousand and ten, DC East (yes I’m in the nations’ capital and it’s lame!), vehicular, or more difficult ones like good morning, my name is, how are you or other common pleasantries.

    She screams and yells. She is hostile about everything. She had been letting us leave at 3 and 3:30 our first two days yet she kept screaming about how far behind “we” are and how “we” are messing her up and how the Census’s computer system crashed and she doesn’t have duplicates of the practice interviews she wants us to do. Some of us have full, and other part-time positions. We were all told it was okay and that working as a numerator for the 2010 Census was easy to work along with another job – full or part-time. Yet our angry and hostile crew leader seems oblivious to these facts. She claims we are to uphold the U.S. Constitution yet there is very little equality or concern for the citizens which make up her crew.

    Her super is just as hostile. He entered our class yesterday while we were on a 15 minute break. Unknown to some of us was that she had changed the time of our break to 7.5 minutes (wtf?). She seems unaware that by law we are to have at least 1 15 minute breakfor every 4 hours (I believe) worked. We all came back within the 15 minutes. He began screaming at us. Never even had the decency to introduce himself to the class before he began his yelling. Blaming us for why she is so far behind. Little did he know that we were on a legitimate break and that she was 15 minutes late that morning – great example she sets for her class.

    It’s all so unprofessional I cannot believe the federal government operates in this manner. Our crew leader is not qualified to teach anything since her communication skills (both reading and speaking) would be at around a 2nd grader’s (forgive me little ones). She was chatting today about what university she attended – thankfully I was seated. I could almost cry.

    I truly wish that on the very first day, when I realized she was incapable of teaching a koala to be cute, I would have requested to be transferred to another class.

    Today, she was just as bad as on the previous 3 training days. She reads like a badly programmed robot operating at a 2nd grade level. She reads EVERYTHING in her manual verbatim. Which includes the crew leader’s (her own) instructions to us. So she will robotically say: Now instruct your trainees to open their work-books to chapter 5 section 3.

    She is a mess and if the U.S. Census was on-line (imagine that!) I think they could have tripled or quadrupled their response totals. It’s ridiculous enough that they hired this moron to teach the enumerator materials shen she cannot speak, read or even do simple math (you should see her doing our circa 1955 time sheets – it’s like Communista Chisanbop – and yet it’s not because her answers are more often than not complettely incorrect.

    Oh the horror!

  13. Trying2MakeSenseof2010Census Says:

    sorry meant to type “enumerator”!