System Failure: “Technology Mayor” Ed Lee’s City “Stuck in 1999”

The City That Knows How to Praise Innovation But Slow to Replace Ancient Software, Organize Data, Reform Municipal IT

By Chris Roberts

April 2, 2012

Technology is never far from Mayor Ed Lee’s mind or lips. With good reason: piles of cash from Silicon Valley luminaries representing Google, Facebook, Salesforce.com and others compelled celebrities like MC Hammer and Brian Wilson to dance for the heretofore obscure former City Administrator and propelled Lee, the onetime interim mayor, to election to a full term as San Francisco’s chief executive last November.

Following his lavishly appointed and well-attended victory party at the Palace Hotel, Lee was quick to signal his appreciation for his friends in the technology sector by gifting Twitter, Inc. a generous tax break to persuade the estimated $8 billion company to not leave the city that “gets it done.”

Board President David Chiu.

Popular tech buzzwords like “entrepreneurship” and “innovation” now spice up Lee’s speech, the latter of which the mayor chose as the subject for his inaugural column for the San Francisco Examiner.

“Innovation is the driver for job creation in San Francisco,” wrote Lee, the mayor of the self-declared “innovation capital of the world,” where 1,500 tech firms employ 30,000 people, just the tip of a technology peninsula that spans from North Point to San Jose.

But there’s a problem for Lee and his image as an innovative tech mayor, one that he inherited from former Mayor Gavin Newsom – the city’s antiquated information technology systems.

“We’re in 2012,” said Board of Supervisors President David Chiu, who sits on the city’s Committee on Information Technology (COIT), a confederation of department heads and bureaucrats tasked with overhauling the city’s IT infrastructure and systems, “but our city’s information technology is stuck in 1999.”

All Gloss and Glamour

There’s no doubt San Francisco’s government has embraced technology. “Twitter Prince” Newsom was one of the social media Web site’s very first celebrities, using the-then startup to announce his aborted bid for California governor (so total was Newsom’s devotion to Twitter that he showed off the company’s Folsom Street HQ to Russian President Dmitry Medvedev during the foreign leader’s 2010 visit to San Francisco).

Newsom insisted city departments become active on social media. Citizens can tweet to DPW when their driveways are blocked or when trash is not picked up, and can similarly publicly query 311 via Twitter – and Newsom used his clout to pass some of the nation’s first open data legislation (though, years on, city departments are asked, not required, to share data with the public).

Chief Innovation Officer Jay Nath.

Under Lee, the city created what it says is the nation’s first “chief innovation officer” position, filled by Jay Nath (a position since copied by other cities), and made much of partnerships with the private sector, intended to get apps created and mine the private sector for ideas meant to “make city government better, more responsive, efficient and connected.”

It is no small wonder Lee looked to the private sector to upgrade San Francisco city government. If San Francisco is a broadband-equipped new Macbook, its city government is trying to get by with a 486 processor-equipped computer using dialup. That is, if they had 486s or dialup modems twenty-eight years ago, which is when some of the software currently in use by city departments was developed and installed.

“Gavin Newsom used to say that he was the Facebook and the Twitter mayor. Ed Lee likes to say he’s the innovation mayor,” said a high-placed city source, speaking candidly on condition of anonymity. “The fact of the matter is, the focus is on the glossy and glitzy aspect of what tech could be. Meanwhile, our IT infrastructure is a mess, and nobody’s focused on it.”

The Mess

The city’s technology infrastructure has long lagged behind the private sector in efficiency, consolidation, and even basic equipment. The San Francisco Police Department made for headline-friendly outrage when it announced that all officers had finally gained access to email – in the summer of 2011. Other reports have detailed the 15-year odyssey of JUSTIS, the still-not-complete technology overhaul of the city’s criminal justice departments, a failure that has been blamed for at least one death.

In 2012, San Francisco city employees use human resources and ledger software developed in the mid-1980s, which the public cannot access, to ascertain line-item budget information; a mishmash of data centers, servers, and computers stuffed in closets house data from payroll information to health records to business and building permits (part of which has only in the last month been backed up to the cloud, a basic first step in Silicon Valley firms), and multiple servers house city workers’ email accounts, all of which were identified as in need of upgrades in city Budget Analyst and Board of Supervisors reports five years ago.

The city also overpays for basic software, with multiple departments paying licensing fees to the same software companies. If the city’s software licenses were consolidated, it is estimated the city could save $50 million – a good chunk of its current $165 million budget deficit.

This makes conducting simple city business more challenging and more expensive, but it also makes city government less open.

One of Lee’s challengers in the November 2011 election, venture capitalist Joanna Rees, tried to make issue of bloated city salaries and pension benefits on the campaign trail. She requested the Mayor’s Office supply line-item budget information, detailing employees’ salaries down to the dollar (as would be available from any of the technology startups Lee’s government swoons over) but she was told no such information was available, not with the way the city’s antiquated human resources system – called FAMIS – stores data.

No One In Charge

Acting Chief Information Officer Jon Walton.

Lee shares at least part of the blame for the city’s state of IT affairs: the city’s Chief Information Officer reports to the City Administrator, Lee’s old job. Former Chief Information Officer Chris Vein oversaw the release of report after report from the City Controller to the Budget Analyst detailing municipal IT’s shortcomings before he left for a job in government in Washington, DC. But in a way, neither Lee nor Vein, nor Acting Chief Information Officer Jon Walton are to blame – because no one person is in charge.

This is common among cities, according to Walton. “Large cities are faced with the common challenge of balancing centralized IT services with decentralized department IT services,” he told Fog City Journal. This means Walton must often rely on departments’ individual goodwill to implement reforms or practices recommended, and accountability for deadlines missed or budgets blown is hard, if not impossible to track. Walton chairs a committee called COIT (Committee on Information Technology), which last year issued the city’s first unified citywide IT plan.

In the Five Year Technology plan released last year, COIT identified ten possible tech upgrades, including a citywide security plan, a citywide dark fiber network, and replacing the telephone system with VOIP (voice over internet protocol) technology. Some of the upgrades mentioned, like a citywide enterprise software agreement – i.e., one license for word processing and data programs instead of multiple licenses – were highlighted in the 2007 report, and have yet to be implemented (translation from tech-speak: five years on, and the city’s still wasting money on multiple licenses for software).

Three major technology upgrades – merging the city’s email platforms, replacing the decades-old HR software, and consolidating and safeguarding the city’s data – are underway. Each has been tasked to a different city department: the Department of Technology, the Controller, and chief innovation officer Nath, respectively.

Email

It’s hard in 2012 to get excited over something as basic as email, yet it’s email consolidation that will be one of the city’s biggest IT accomplishments this year.

Years ago, multiple city reports addressed the need for a consolidated email platform, with some city users on LotusNotes, and others on Microsoft Exchange. Today, “I still cannot easily e-mail a city employee,” said Chiu, who named seven different city email servers.

This is at last changing. The city’s email consolidation project is “well underway,” Walton wrote in an email, with about 1,000 city users on a consolidated Microsoft Exchange platform. That leaves about 20,000 users left to go, however, according to the most recent information provided by Walton in response to an email request. Some city departments have yet to be scheduled for a “go-live date,” and no users were able to import their contacts, attachments, or calendaring, according to a city report.

E-Merge

Close to one-third of San Francisico’s $6.8 billion budget goes to city employees. The software used to manage the city’s 22,000 employees is 15 years old, a spring chicken compared to the software used to watch over the city payroll, which is 28 years old, according to the Controller’s Office, which is tasked with the $45 million consolidation/replacement of the ancient software, an undertaking dubbed “E-Merge.”

Deputy Controller Monique Zmuda.

Replacing the vintage computer systems has thus far taken four years, according to Deputy Controller Monique Zmuda, in part because of software customization by city programmers. The new system will manage “all human resources functions, payroll, benefits administration, and time reporting,” Zmuda said.

Still to be done is the replacement of FAMIS, the city’s financial accounting system that is entering its 25^th year of service. The cost of this project is unknown as well as the total scope, in part because “many departments utilize their own small sub-systems to compliment the city’s central accounting system,” Zmuda wrote.

Solving that lack of centralization and putting the city’s data under fewer roofs has been tasked to Nath.

There’s not much room for public visibility of city data, yet. The Controller plans to make available to the public more budgetary data, but to do so in a way that’s “understandable and readable” is tricky, Zmuda said. “The trick is to modify the old system to enable the data to be reformatted and re-sorted.”

Efforts to put the city’s budgeting data in a publicly-available portal were initially dismissed because it would be “difficult,” according to a city source. Difficult, and more under the purview of the city’s data initiative, which is asking – politely – for city departments to make data available to the public on a Web site.

Data

Lee and Nath announced in early March that the city had moved some of its data from local servers and hard drives to the cloud. The city’s open data is also the envy of the rest of the world, Nath told Fog City Journal. With “200 datasets” online, and over “60 apps” developed, “we’ve been an international leader on open data,” Nath wrote via e-mail.

Yet private companies would still find much lacking in the way the city houses its data, in a mélange of data centers, data closets and stand-alone servers. A corporation or another comparable organization should have “two, at most maybe three” data centers, according to a high-placed city source. “Instead, we have three dozen.”

San Francisco is budgeted to spend $7 million over the next two years to shore up this situation, with “900+” data locations consolidated into two servers at 200 Paul Avenue, a privately-owned data center in the city’s Bayview District, according to Walton, with another city data center located at the airport. The Bayview data center is scheduled to be built by November, with city departments to move their IT servers to the data center in about a year from now, according to documents provided by Walton.

The city’s data is still susceptible to a major disaster. In the event of a major earthquake, for example, city workers would be unlikely to have access to email – and there are no plans to consolidate the city’s data into a user-friendly “data warehouse” akin to what was done at the federal level with Data.Gov, according to Nath. “It’s proved expensive [for other cities],” he said, “and doesn’t really bring us to where we need to be.”

What is available on Data.sfgov.org are datasets in the form of spreadsheets listing all licensed San Francisco businesses, for example. While it’s perhaps not likely a Microsoft Excel file would excite a Facebook executive at an innovation meeting, this is a big step for “transparency” in local government, Nath said. “Just because it’s not the most cutting edge approach, doesn’t diminish the inherent value of the information.”

“There is no simple way to just push all data to the public – this takes time,” said Nath, who told a reporter, “If you have ideas on strengthening legislation, I’m open to hearing it.”

Leadership

Calls to centralize the city’s IT systems were made as long ago as 2001. “Currently there is no formal, or dotted line reporting relationship between the City CIO and the various department CIO’s or technology staff,” the Department of Technology wrote in 2007. This remains true today.

No one person or department is in charge of the city’s necessary IT upgrades – many city departments have their own CIOs and own IT staff. This arrangement not only duplicates roles and personnel, it makes citywide reform very hard to implement. The city’s IT is run like a “confederation,” according to one insider, and ostensible leaders like Walton must depend on department’s good will in order to ensure their cooperation.

The city has a Department of Technology, but it has no sway over many of the city’s IT workers, decisions, and infrastructure. That’s left to individual city departments’ information officers and IT staff. No one person has control over the one main command-making body, COIT, which meets once a month for about an hour and a half, according to records.

COIT’s main function is to provide “policy and guidance,” and most city tech projects do go to COIT for approval, said Ron Vinson, who heads up the Department of Technology. Often COIT provides little more than a rubber stamp, said one former city employee.

Information technology does not appear to be a priority, at least from a budgeting perspective. Of the city’s $6.8 billion budget, a little more than $40 million was dedicated to new IT projects, such as upgrading the city’s 911 dispatch center to the latest software ($1.2 million alone).

Meanwhile, the city’s overall IT budget has shrunk. The city spent $173 million on information technology between 2006-2007, according to a 2007 Budget Analyst report.  In the current fiscal year, the city is spending $160 million, according to Walton, whose own position as Chief Information Officer means he chairs COIT, but does not hold sway over the Department of Technology. Walton is also still “acting” Chief Information Officer (newly-minted City Administrator Naomi Kelly is tasked with making a permanent hire; her office did not respond to telephone calls seeking comment).

Insiders have before called for COIT’s dissolution, to get the city to move more quickly and efficiently, to get San Francisco to “take a forward-thinking approach… responsive to the ever changing industry of information technology.” In other words, “innovation.”

Those words are lifted directly from the 2007 Budget Analyst’s IT report which suggested reforms still waiting to be implemented. In the meantime, without a true structural shakeup or reform, it’s proven difficult and slow to innovate a city IT bureaucracy the size of San Francisco’s.

“At the end of the day, someone at the top needs to make sure these projects get done, no excuses or this bureaucratic stuff,” one insider said. “People are going to help, and if they can’t, they ought to be fired or be demoted.” That, at least by today’s standards, would be innovative.

Chris Roberts

Just trying to make a little trouble.

More Posts