My Two Census

Formerly the non-partisan watchdog of the 2010 US Census, and currently an opinion blog that covers all things political, media, foreign policy, globalization, and culture…but sometimes returning to its census/demographics roots.

Posts Tagged ‘Regional Directors’

Anatomy of a Paper-Based Operations Control System (PBOCS) failure…

Wednesday, April 21st, 2010

Below are e-mails obtained by MyTwoCensus.com sent from Brian Monaghan and Barbara M. Lopresti at Census Bureau Headquarters to every regional Census Bureau office in America that describe IT systems failures:

From:
Brian Monaghan/FLD/HQ/BOC

To:

FLD Regional Directors

Cc:

FLD Deputy Regional Directors List, Barbara M LoPresti/TMO/HQ/BOC@BOC, Chad G Nelson/TMO/HQ/BOC@BOC, Janet R Cummings/FLD/HQ/BOC@BOC, Gail A Leithauser/FLD/HQ/BOC@BOC, Marilia A Matos/DIR/HQ/BOC@BOC, Annetta Clark Smith/DMD/HQ/BOC@BOC, Michael T Thieme/DMD/HQ/BOC@BOC, Timothy P Olson/FLD/HQ/BOC@BOC

Date:

04/20/2010 09:52 AM

Subject:

PBOCS and NRFU

OK, folks…   here’s where we are as of 8:45 a.m. Tuesday morning  -

As of 6:00 p.m. or so Monday evening, the last of the LMR automated removal occurred.  On Friday, April 23, there will be a PBOCS deployment which will include the reports of LMRs since Monday.  Those reports will then be available for clerical line-through of LMRs on the assignment registers (which are hopefully being printed by then).

We are expecting all of the even numbered AAs to have their reports (listings, labels, etc.) generated in the system by 11:00 a.m. this morning.

The system continues to be somewhat unstable, so at midnight tonight we need all LCOs and RCCs to get off PBOCS and stay off until Thursday morning (we hope).  That will give us a clean opportunity to generate the majority of reports for the odd numbered AA’s (we hope).  So…  no users on the system starting at midnight tonight and lasting through Wednesday.

Our # 1 priority is to get all of the reports generated and copied to an alternative printing site, so that if PBOCS goes down, the LCOs will still be able to print materials needed for NRFU assignment prep.   Once the even numbered AAs have all of their reports generated (by 11:00 a.m. this morning), we will begin the process of exporting the files to an alternative print site.   Several additional meetings need to occur to work through all of the details, but our hope is that DOTS will be testing this alternative printing site in, say, one LCO per region…  ideally nearby the RCC so your LSC can observe…  either this afternoon or tomorrow morning.  It’s not clear at this point whether we will be able to pull everything together that quickly.

Bottom line is that we are still planning for the LCOs to begin printing assignments for the even numbered AAs Thursday morning…  either through PBOCS or the alternative print site.  At this time, we are assuming all other PBOCS users will also regain access to the system Thursday morning.   We have asked that odd numbered AAs be made available on a flow basis of some sort…  groups of LCOs or regions…  rather than waiting until all reports are generated to make them available for printing. This weekend will be a huge crunch time for the LCOs…  all hands need to be on deck…  as they prepare assignments for all of the even numbered AAs and as many of the odd as possible.

Please make sure the LCOs are firing on all cylinders with NRFU map printing. That task is outside of PBOCS, so the downtime tomorrow will not be a problem.  It’s really critical to get this job done ASAP, so that the printers in the LCO are not tied up with NRFU maps, and are available for assignment prep.    If you cannot get all NRFU maps done by COB Wednesday, give top priority to the even numbered AAs, so assignment prep can be completed for work headed to the field first thing next week.  An added impetus to the NRFU map printing work is that there is a remote chance that LCOs may be able to start assignment prep for even numbered AAs tomorrow (Wednesday) if we are able to get the alternative print site set up, files exported, systems tested in some LCOs, and instructions prepared.   LCOs which have completed NRFU map printing will be likely candidates for this somewhat unlikely event.

We can talk more at the RD Conference Call this afternoon, or call me if you have an immediate concern.

From:
Brian Monaghan/FLD/HQ/BOC

To:

FLD Regional Directors

Cc:

FLD Deputy Regional Directors List, FLD Decennial Branch Chiefs, FLD Decennial Assistant Division Chiefs List, Marilia A Matos/DIR/HQ/BOC@BOC, Barbara M LoPresti/TMO/HQ/BOC@BOC, Michael T Thieme/DMD/HQ/BOC@BOC, Chad G Nelson/TMO/HQ/BOC@BOC

Date:

04/14/2010 09:48 AM

Subject:

PBOCS

As you may know, PBOCS went down last night.  The 40 LCOs that were scheduled to be ingested did not get ingested.  PBOCS is back up this morning and available for your use, but the concerns about instability remain.

We must do the following to prepare for NRFU:

PBOCS will be taken down tonight at 8:00 p.m., and will not be available again until Monday morning, April 19. Hopefully, minimizing the number of users and uses will increase the stability of the system, allowing the full ingest of all LCOs to be completed over the next several days.  As you heard at the Regional Directors’ Conference, this is a critical first step in the process of preparing for NRFU assignment prep.

DOTS will be sending out a separate notice to you and your automation folks, and each of the Decennial Branch Chiefs will issue ops logs with suggestions and cautions about getting through the next several days.  For example,  it’s critically important not to send completed work to the processing office unless it has been checked out through PBOCS.  If you box up and send in ICRs/MCRs without going through the formal PBOCS check-out process, we will lose the critical linkage with their Group Quarters.   We will be asking you to hold completed work in the office until PBOCS is back up and running.  Of course, work on all operations can and should continue in the field.

This will be a really important time for the LCOs to stay as organized and systematic as possible…   labeling and sorting piles of completed and pending work in  a way such that, when PBOCS is made available, we can rapidly recover.  If work needs to go to the field while PBOCS is down, the LCOs will need to manually track the assignments, so they know who has what, and when  they got it.  Once PBOCS is made available on Monday, the LCOs will need to key in this information to get the system caught up.

Thanks for your patience as we work through these challenges.

From:
Brian Monaghan/FLD/HQ/BOC

To:

fld.regional.directors@census.gov

Cc:

fld.decennial.assistant.division.chiefs.list@census.gov” <fld.decennial.assistant.division.chiefs.list@census.gov>, “Barbara M LoPresti” <barbara.m.lopresti@census.gov>, “Chad Nelson” <chad.g.nelson@census.gov>, “fld.deputy.regional.directors.list@census.gov” <fld.deputy.regional.directors.list@census.gov>

Date:

04/08/2010 05:57 PM

Subject:

Fw: PBOCS System Outage starting Friday April 9th at 500pm ET.

We need to shut down PBOCS at 5:00 p.m. Friday, April 9, instead of waiting until midnight.  We had a lengthy discussion today and, as you can imagine, time is a critical commodity.  Lots of work has to be done in preparation for NRFU, and if it means an extension for ETL or delays in check-in of GQE and UE, so be it.

Call me if you have any questions or just need to vent.  We wouldn’t be doing this full weekend shutdown if it wasn’t really necessary.

    Inactive hide details for Barbara M LoPresti Barbara M LoPresti

—– Original Message —–
From: Barbara M LoPresti
Sent: 04/08/2010 05:02 PM EDT
To: Brian Monaghan; Chad Nelson; Janet Cummings; Gail Leithauser; Annetta Smith; Michael Thieme; Pamela Mosley; Marilia Matos; Arnold Jackson
Cc: Thomas McNeal; Curtis Broadway
Subject: PBOCS System Outage starting Friday April 9th at 500pm ET.
Brian,
In the 430 meeting today, Tom and Curtis felt it was best to take the Pbocs system down at 5::00 pm eastern time on Friday, April 9th (tomorrow).
Please let me and Chad know when you have informed the RDs and then we will get a DOTS message out to the RCCs.
Thanks
Barbara