Wednesday, October 3, 2018

Recap: July 2016 - Nov 2017

UCPath Implementation


As the MFA project moved into production my role diminished and I was assigned to the UCPath project to serve as the Technical Lead for the IdM workstream.  UCPath is a UC-systemwide Payroll and Human Resources system intended to replace the 30+ year old mainframe PPS (Payroll Personnel System). UCPath is an acronym that stands for UC Payroll, Academic Personnel, Timekeeping, and Human Resources. UCLA ran an instance of PPS for UCLA, UCOP, ASUCLA, and UC Merced, and UCOP ran an instance of PPS for other campuses. The project was announced in October 2009 under UC President Yudof, with PeopleSoft selected in 2011. The Office of the President (UCOP) was the first to go live in December 2015. The next wave - termed the "pilot" - was targeted for August 2017 and was to include the campuses UCLA, UC Riverside, and UC Merced, and the organization ASUCLA.

Because UCOP utilized several of UCLA's Identity related systems, even though UCLA was not on UCPath there were integrations set up between UCPath and the UCLA systems so that as new employees were hired at UCOP they would be assigned their UCLA ID (UID) from the UCLA UID system and that ID communicated to the UCOP IdM system and UCPath. These integrations went live in December of 2015 but did not get much use because of the low volume of employees at UCOP. Also there were concerns that some decisions had been made to meet the UCOP go-live deadline that might require rethinking in order to scale to support UCLA.

While much work had been done to meet UCOP requirements, there remained much to do in order to fully support UCLA. Because of the number of separate test phases - system test, integration test, DR test, QA test - the time allowed for development and developing thorough test cases became increasingly compressed and the time for correction of defects discovered in testing cycles was reduced. Eventually the decision was made to delay the pilot go-live date to December 2017. By July 2017 testing was proceeding well but there remained significant concerns with the quality of the data and other aspects of the project. In early October UCLA announced that it was dropping out of the pilot group and would defer until 2018. ASUCLA, UC Merced, and UC Riverside decided to proceed on schedule, and because ASUCLA and UC Merced were heavily dependent on UCLA Identity services all of the IdM integration work still needed to be completed and implemented on schedule, so the decision for UCLA to defer actually added tasks to our workstream rather than reducing them. By that time, because the IAMUCLA team needed to take ownership for production support and I officially started in the role of IT Architecture supervisor on November 1 my role as IdM workstream lead was turned over to the IAMUCLA lead. I continued working as an SME with the project through the holidays but by the time ASUCLA, UC Merced, and UC Riverside officially went live on January 2, 2018 I was no longer heavily involved.

There was a wave of departures, retirements, and reassignments that started in January 2018 and continued through the summer, including our CIO and several of the IT directors. One can only speculate that the 11th hour decision to defer UCPath for UCLA was a contributing factor in some cases.


UCLA went live on UCPath on September 23, 2018 along with UC Santa Barbara. It is not widely understood that, due in part to the efforts of my team from July 2016 to November 2017, that by that time many of the IdM processes and integrations had been running in production for nearly a year.

And Is That All?


Though the UCPath project did take up the bulk of my time I did have the opportunity to participate in a few other major activities.

Student System Replacement RFP

In 2016 an effort was underway to consider replacing the Student System and I began participating in March 2016. In April I began chairing the RFP Access & Roles Definitions sub-group which developed that section of the formal RFP. The sub-group effort concluded in November though the materials we generated were then used as input for similar sections in the Financial System Replacement RFP which was spinning up around that time.

Financial System Replacement RFP

In late 2016 the Financial System Replacement effort began with the development of an RFP. The sections of the Student System Replacement RFP that my sub-group had developed were leveraged for the Financial System Replacement RFP. My role was primarily to review and edit those sections of the final RFP accordingly.

Enterprise IT Architecture Team

An Enterprise IT Architecture team had been created within IT Services by the Associate Vice Chancellor in 2012. An optimistic and visionary roadmap was developed at that time and training was done, but over the years the director had resigned and the group first lost focus and gradually influence. In 2016 Albert Wu, a senior director in IT Services, took ownership of the team and sought to assess and rebuild. Since I was his IdM Architect he added me into the team. Leadership of the team was rotated quarterly and I served a stint as chair, but my primary focus was to observe and evaluate and consider areas of potential improvement for both the team and IT Services. This lasted through the fall of 2017. At that time I applied for the permanent position as supervisor of the team and officially began in that role on November 1st. I was still engaged with UCPath which required implementation of the IdM interfaces for UCOP, ASUCLA, and UC Merced and I continued to remain involved in that through the holidays.

The EITARCH team during 2017 was focused primarily on establishing the groundwork for future architecture work, with a special focus on Business Architecture and governance processes.

There were many opportunities during this year to establish relationships with individuals from a variety of UCLA departments, many of whom were business process owners. From that perspective it was a rewarding year.

Recap: Oct 2015 - July 2016

UCLA Campus Multi-Factor Authentication


In the Beginning

MFA High-level Decision Diagram - B. Bellina, Nov 2015
Most of my first year at UCLA I was dedicated to the development and implementation of the campus Multi-Factor Authentication solution. Of course MFA was nothing new to Higher Ed and presentations going back to 2011 were readily available (IAMOnline has several). UCLA had adopted hardware tokens at one point in the past, not to great acclaim, and MFA was not seriously considered until 2015 when it was proposed by Albert Wu following a security event. It was then adopted as an Information Security program under the interim director of IT Security Michael Story. At the time I was hired in late 2015 the Duo product for MFA had been selected and the campus Shibboleth Single Sign-On (SSO) implementation was being considered as the first major implementation for the campus to adopt MFA, with other services such as department VPN's to follow. I had practical MFA experience from my time at USC. In 2013 while managing IdM at USC, I presented at the EDUCAUSE Annual conference on the topic and a year before Russell Beall on my team had developed and presented a functional prototype of a groups-driven Shibboleth SSO implementation with Duo for MFA. The prototype was shelved until 2015 when, after an Information Security leadership change, it was resuscitated and my former team at USC, now under Asbed Bedrossian's leadership, quickly productionalized the prototype and implemented MFA with Shibboleth, first rolling it out to 570 IT and critical staff and then expanding to all 15,000 staff on December 1, 2015.

Shibboleth Login MFA flow for UCLA - B. Bellina, Nov 2015
Although some investigative work at UCLA was done in the final quarter of 2015, the official "Multi-Factor Authentication" project kick-off was January 27, 2016. The goal was to have a functional MFA solution developed by Memorial Day running on the Shibboleth Identity Provider v3. Our currently implemented Shibboleth IdP v2 was going to be end-of-lifed in June 2016 along with SAML 1 support so we had to plan the IdP v3 upgrade and migrate all SAML 1 SP's to SAML 2 as well as integrate MFA. After a QA period we would then upgrade the IdP begin controlled and optional enrollment of MFA in May. Considering the complexity of the UCLA environments it was an aggressive schedule but the fixed end-of-life of Shibboleth IdP v2 made this a firm deadline. We held discussions with the IdM teams at both USC and Penn State to discuss their MFA Duo implementations. At that point Penn State was closing in on 10,000 enrolled staff/faculty MFA users with plans to enroll all 25,000 faculty and staff by May. While documenting our requirements we considered the work done elsewhere, all of which had been done for IdP v2: Duo's own Shibboleth IdPv2 plug-in, a Duo MFA on demand IdP v2 solution proposed by Russ Beall here <https://wiki.shibboleth.net/confluence/display/SHIB2/Duo+2FA+On+Demand>, and the IdP v2 Shibboleth Multi-Context Broker. In the end we contracted with Unicon and John Gasper worked with us to develop our own custom IdP v3 logic that leveraged the standard eduPersonEntitlement attribute which we populated from Grouper groups. It was tight but the compiled code was turned over by Memorial Day and even included the hooks needed to support the AuthnContextClassRef profile for MFA being proposed by InCommon.

B. Bellina, Feb 2016

Talking the Talk

Along the way, in addition to developing technical requirements, working with Unicon, and getting the test and QA environments set up, there needed to be continuous communication to campus groups to belay fears, avoid misunderstandings, develop partnerships, and request feedback. Often these communications were in the form of presentations and because of my experience in that area I had several opportunities in the months between January and June to both contribute to presentations and give them. It was for one of the early presentations that I applied the tagline "As Simple as 1... 2... 3" which in one form or another continued to be used in UCLA MFA literature for years afterward.

A partial list of presentations given in 2016 at UCLA include:
  • Jan 12 - IAMUCLA Townhall
  • Feb 17 - UCLA Info Sec Ask the Experts "Multi-Factor Authentication"
  • Mar 8 - IAMUCLA Townhall
  • Mar 22 - Common Systems Group meeting - "Multi-Factor Authentication"
  • Apr 14 - DIIT Spring Staff Meeting "IT Services Information Security Program"
  • Apr 26 - Common Systems Group meeting - "Deploying Multi-Factor Authentication with UCLA Logon"
  • Apr 27 - BruinTech Tech-a-thon "Multi-Factor Authentication: The UCLA Campus Service"
  • May 3 - IAMUCLA Townhall
  • Jun 27 - DACSS Training half-day seminar "Multi-Factor Authentication"
  • Nov 17 - BruinTech Brown Bag "Multi-Factor Authentication"
There were also opportunities in 2016 and even 2017 to present to larger audiences as well at UC, InCommon, and Internet2 events including:

Walking the Walk

The MFA production rollout began on time in June 2016 with 9 enrolled users on June 6 performing 40 MFA-enabled logons a day. By June 29 there were 129 enrolled users performing over 700 MFA-enabled logons per day. My time on the project ended in July but the rollout continued to the campus community and eventually enhancements were done to the enrollment user interface.
  • By end of 2016 there were over 700 users enrolled.
  • By end of June 2017 there were over 2,500 users enrolled.
  • October 31, 2017 all non-medical faculty and staff are required to use MFA to access campus applications through SSO and campus VPN, increasing enrollment to over 26,000.
  • By end of 2017 there were over 31,000 users enrolled performing over 50,000 MFA-enabled logons per day.
  • April 17, 2018 all students are required to use MFA to access campus applications through SSO and campus VPN, increasing enrollment to over 71,000 and performing over 100,000 MFA-enabled logons per day.
Because all incoming UCLA employees and students are now mandated to use MFA in order to access web applications, including the new payroll system, the numbers continue to rise. At this time UCLA is undoubtedly one of the largest Duo MFA implementations of any university in the United States.

This was a project I am proud to have been a part of and I remain grateful to Albert Wu for giving me the opportunity.