My Two Census

Formerly the non-partisan watchdog of the 2010 US Census, and currently an opinion blog that covers all things political, media, foreign policy, globalization, and culture…but sometimes returning to its census/demographics roots.

Posts Tagged ‘Paper Based Operational Control System’

Anatomy of a Paper-Based Operations Control System (PBOCS) failure…

Wednesday, April 21st, 2010

Below are e-mails obtained by MyTwoCensus.com sent from Brian Monaghan and Barbara M. Lopresti at Census Bureau Headquarters to every regional Census Bureau office in America that describe IT systems failures:

From:
Brian Monaghan/FLD/HQ/BOC

To:

FLD Regional Directors

Cc:

FLD Deputy Regional Directors List, Barbara M LoPresti/TMO/HQ/BOC@BOC, Chad G Nelson/TMO/HQ/BOC@BOC, Janet R Cummings/FLD/HQ/BOC@BOC, Gail A Leithauser/FLD/HQ/BOC@BOC, Marilia A Matos/DIR/HQ/BOC@BOC, Annetta Clark Smith/DMD/HQ/BOC@BOC, Michael T Thieme/DMD/HQ/BOC@BOC, Timothy P Olson/FLD/HQ/BOC@BOC

Date:

04/20/2010 09:52 AM

Subject:

PBOCS and NRFU

OK, folks…   here’s where we are as of 8:45 a.m. Tuesday morning  -

As of 6:00 p.m. or so Monday evening, the last of the LMR automated removal occurred.  On Friday, April 23, there will be a PBOCS deployment which will include the reports of LMRs since Monday.  Those reports will then be available for clerical line-through of LMRs on the assignment registers (which are hopefully being printed by then).

We are expecting all of the even numbered AAs to have their reports (listings, labels, etc.) generated in the system by 11:00 a.m. this morning.

The system continues to be somewhat unstable, so at midnight tonight we need all LCOs and RCCs to get off PBOCS and stay off until Thursday morning (we hope).  That will give us a clean opportunity to generate the majority of reports for the odd numbered AA’s (we hope).  So…  no users on the system starting at midnight tonight and lasting through Wednesday.

Our # 1 priority is to get all of the reports generated and copied to an alternative printing site, so that if PBOCS goes down, the LCOs will still be able to print materials needed for NRFU assignment prep.   Once the even numbered AAs have all of their reports generated (by 11:00 a.m. this morning), we will begin the process of exporting the files to an alternative print site.   Several additional meetings need to occur to work through all of the details, but our hope is that DOTS will be testing this alternative printing site in, say, one LCO per region…  ideally nearby the RCC so your LSC can observe…  either this afternoon or tomorrow morning.  It’s not clear at this point whether we will be able to pull everything together that quickly.

Bottom line is that we are still planning for the LCOs to begin printing assignments for the even numbered AAs Thursday morning…  either through PBOCS or the alternative print site.  At this time, we are assuming all other PBOCS users will also regain access to the system Thursday morning.   We have asked that odd numbered AAs be made available on a flow basis of some sort…  groups of LCOs or regions…  rather than waiting until all reports are generated to make them available for printing. This weekend will be a huge crunch time for the LCOs…  all hands need to be on deck…  as they prepare assignments for all of the even numbered AAs and as many of the odd as possible.

Please make sure the LCOs are firing on all cylinders with NRFU map printing. That task is outside of PBOCS, so the downtime tomorrow will not be a problem.  It’s really critical to get this job done ASAP, so that the printers in the LCO are not tied up with NRFU maps, and are available for assignment prep.    If you cannot get all NRFU maps done by COB Wednesday, give top priority to the even numbered AAs, so assignment prep can be completed for work headed to the field first thing next week.  An added impetus to the NRFU map printing work is that there is a remote chance that LCOs may be able to start assignment prep for even numbered AAs tomorrow (Wednesday) if we are able to get the alternative print site set up, files exported, systems tested in some LCOs, and instructions prepared.   LCOs which have completed NRFU map printing will be likely candidates for this somewhat unlikely event.

We can talk more at the RD Conference Call this afternoon, or call me if you have an immediate concern.

From:
Brian Monaghan/FLD/HQ/BOC

To:

FLD Regional Directors

Cc:

FLD Deputy Regional Directors List, FLD Decennial Branch Chiefs, FLD Decennial Assistant Division Chiefs List, Marilia A Matos/DIR/HQ/BOC@BOC, Barbara M LoPresti/TMO/HQ/BOC@BOC, Michael T Thieme/DMD/HQ/BOC@BOC, Chad G Nelson/TMO/HQ/BOC@BOC

Date:

04/14/2010 09:48 AM

Subject:

PBOCS

As you may know, PBOCS went down last night.  The 40 LCOs that were scheduled to be ingested did not get ingested.  PBOCS is back up this morning and available for your use, but the concerns about instability remain.

We must do the following to prepare for NRFU:

PBOCS will be taken down tonight at 8:00 p.m., and will not be available again until Monday morning, April 19. Hopefully, minimizing the number of users and uses will increase the stability of the system, allowing the full ingest of all LCOs to be completed over the next several days.  As you heard at the Regional Directors’ Conference, this is a critical first step in the process of preparing for NRFU assignment prep.

DOTS will be sending out a separate notice to you and your automation folks, and each of the Decennial Branch Chiefs will issue ops logs with suggestions and cautions about getting through the next several days.  For example,  it’s critically important not to send completed work to the processing office unless it has been checked out through PBOCS.  If you box up and send in ICRs/MCRs without going through the formal PBOCS check-out process, we will lose the critical linkage with their Group Quarters.   We will be asking you to hold completed work in the office until PBOCS is back up and running.  Of course, work on all operations can and should continue in the field.

This will be a really important time for the LCOs to stay as organized and systematic as possible…   labeling and sorting piles of completed and pending work in  a way such that, when PBOCS is made available, we can rapidly recover.  If work needs to go to the field while PBOCS is down, the LCOs will need to manually track the assignments, so they know who has what, and when  they got it.  Once PBOCS is made available on Monday, the LCOs will need to key in this information to get the system caught up.

Thanks for your patience as we work through these challenges.

From:
Brian Monaghan/FLD/HQ/BOC

To:

fld.regional.directors@census.gov

Cc:

fld.decennial.assistant.division.chiefs.list@census.gov” <fld.decennial.assistant.division.chiefs.list@census.gov>, “Barbara M LoPresti” <barbara.m.lopresti@census.gov>, “Chad Nelson” <chad.g.nelson@census.gov>, “fld.deputy.regional.directors.list@census.gov” <fld.deputy.regional.directors.list@census.gov>

Date:

04/08/2010 05:57 PM

Subject:

Fw: PBOCS System Outage starting Friday April 9th at 500pm ET.

We need to shut down PBOCS at 5:00 p.m. Friday, April 9, instead of waiting until midnight.  We had a lengthy discussion today and, as you can imagine, time is a critical commodity.  Lots of work has to be done in preparation for NRFU, and if it means an extension for ETL or delays in check-in of GQE and UE, so be it.

Call me if you have any questions or just need to vent.  We wouldn’t be doing this full weekend shutdown if it wasn’t really necessary.

    Inactive hide details for Barbara M LoPresti Barbara M LoPresti

—– Original Message —–
From: Barbara M LoPresti
Sent: 04/08/2010 05:02 PM EDT
To: Brian Monaghan; Chad Nelson; Janet Cummings; Gail Leithauser; Annetta Smith; Michael Thieme; Pamela Mosley; Marilia Matos; Arnold Jackson
Cc: Thomas McNeal; Curtis Broadway
Subject: PBOCS System Outage starting Friday April 9th at 500pm ET.
Brian,
In the 430 meeting today, Tom and Curtis felt it was best to take the Pbocs system down at 5::00 pm eastern time on Friday, April 9th (tomorrow).
Please let me and Chad know when you have informed the RDs and then we will get a DOTS message out to the RCCs.
Thanks
Barbara


Anonymous Census Bureau Official: Major Nation-wide IT FAILURE at the Census Bueau

Monday, April 19th, 2010

The following piece comes to us from a Census Bureau official from the Mid-West region who has requested anonymity but has had her identity verified by MyTwoCensus.com. Her discussion of a major week-long IT systems failure has effected every Census Bureau office across the country and demonstrates how disgraceful operations are being handled (and subsequently covered up, as MyTwoCensus has requested information about this failure numerous times yet has not received any responses from the Census Bureau’s Public Information Office):

I have a problem in my local census office. The problem is 493 of those offices across the nation are having the same exact problem. You see, in November and December 2009 when I first started at the Census Bureau, one of my first assignments was assisting in a system load test known as the Decennial Application Load Test (or DALT for short). DALT simulated the conditions of every census office employee logging on at the same time across the nation and what would happen. This was my first taste of our federal tax dollars hard at work. As is typical of any large scale government program the whole system went kaput; users couldn’t log-on and applications crashed.

The Bureau’s contingency plan was to limit users and ask offices to spread out their staffing. We started to run an evening shift which meant paying night differential (an additional 10% pay) to our night employees.  One application of concern was PBOCS the Paper Based Operational Control System that was unveiled in January 2010 to complete the rest of their field operations. The system was developed after the Census decided to nix the Harris Corporation as their contractor. The PBOCS system was used to check out assignments into the field, make assignments to enumerators, check work into the office after completion and check out for shipping. In February 2009 after the two big snowstorms that hit Washington DC the same thing happened. The Bureau’s contingency plan for PBOCS was to limit users to four per local census office and remove functionality they didn’t deem was mandatory.

In the last few weeks we are experiencing many more problems as we near non-response followup the most massive operation of the decennial census. Some of PBOCS problems include but are not limited to:
* individual census reports (ICRS) from group quarters enumeration that PBOCS claims are missing for shipment.
* daily progress reports that are outdated or showing the wrong numbers
* numerous system crashes where work is lost or has to be rescanned
* wrong or missing work being selected for reinterview

At the beginning when these problems started, the RCC pointed fingers at the local census offices accusing them of not using the system or processing the work properly. However when several offices reported the same issues the blame then shifted to the software.

To further complicate things is the divide that also occurs between the local census office employees and the the regional census center and headquarters staff. The headquarters and regional census center staff are mostly career employees. The local census office employees are temporary Schedule A hourly employees. Although both are referencing the same procedures in the same manuals the local census office employees are the ones who are doing the grunt work (the enlisted men per say). The RCC and headquarters staff (the commissioned officers) manage and oversee but do not realize the difficulties and nuances because they are not out there getting their hands dirty.

The LCO employees are finding that the PBOCS system actually will not update the report numbers or sometimes show the wrong numbers. The RCC and headquarters are lead to believe if the work doesn’t show in the system as completed then it is not completed. However when the reports don’t show the quotas are being met the career census employees usually get on the phone to threaten the temporary employees and even sometimes terminating their employment.

This weekend is the third weekend PBOCS has gone offline. On Wednesday April 14 it was announced PBOCS was to go down at 8pm (actually went down about 5pm) and will not be back up until Monday morning April 19th. We are somewhat relieved to hear through the grapevine that the RCC and headquarters will be more lenient and readjust their production goals for the entire nation. However I think that much of the intimidation and harassment will occur again so that the career census employees can cover their asses and recover any of the money they were probably paying in overtime for programmers to fix this crap piece of software. This is all the while they tell the temporary employees at the LCO that overtime will be strictly prohibited.

The career employees at RCC and headquarters hold the temporary LCO employees accountable for mistakes they [RCC and headquarters] made in the ten years they had to prepare since the last decennial. But who is held accountable at the highest levels when a multi-billion dollar piece of software doesn’t work and they constantly have to fix it? I hope the Inspector General and my congressmen are reading this because PBOCS is government waste at its finest. How ironic it is happening in the nearing days of the largest operation of the decennial census.