APS is continuing to work to restore power to affected buildings along the Capitol Mall. Buildings along Adams seemed to have been restored and buildings along Jefferson are still out.
We will post again when we have more information.
APS is continuing to work to restore power to affected buildings along the Capitol Mall. Buildings along Adams seemed to have been restored and buildings along Jefferson are still out.
We will post again when we have more information.
We are aware of a power outage affecting most of the Capitol Mall area. APS is reporting their field personnel have been dispatched to investigate an unknown interference to a major power line. All buildings along the Capitol Mall have lost power, which means that lights, Internet, A/C and elevators are not currently operating. It is also affecting the street lights in the area. There is no guaranteed repair time, but APS is estimating restoration sometime between 1:30-2:30pm today.
We will provide another update at 12:30PM.
Talent and Acquisition is back up and working, sorry for any inconvenience.
At this time we are experiencing issues with HRIS TA, we have our Technical team investigating and we will provide and update with in the hour.
This outage has been resolved through the hard work of the ASET team and vendor counterparts to identify the router that was causing the issue, disable it and restore affected agencies to normal operational functions. The disabled router has been repaired and placed back into service as of 2:00pm. The router and customer locations are being closely monitored for the rest of the business day to ensure services stay up and active.
Some agencies have been experiencing intermittent outages with phones and Internet at offices throughout the State. The ADOA ASET team is working closely with CenturyLink and initial indications are that it is a statewide outage on the carrier network, affecting their Metro Optic Ethernet service. AZNet has confirmed the following agencies are affected: ADC, ADOT, DCS, DES and ADJC. This does not appear to be weather related. There is no estimated time for repair. We will update when more information is known.
Phone and internet has been restored to affected agencies. Thank you to all the vendors and ADOA teams that worked diligently to resolve this issue.
Update: 12:45pm
Centurylink has identified the break in the cable fiber line and pinpointed its location. Technicians are working diligently to address the problem and estimate services to all agencies will be restored by 2:00 p.m. today. All services offered by the Arizona Department of Transportation, including MVD offices and ServiceArizona.com, are now up and operational. Phone and internet services remain affected for the following agencies:
Dept of Administration
Dept of Revenue
Dept of Agriculture
Industrial Commission of Arizona
Corporation Commission
Dept of Juvenile Corrections
Dept of Economic Security
Dept of Real Estate
Dept of Insurance
Commission for Deaf and Hard of Hearing
Centurylink experienced a break in a fiber cable that is impacting some state services. Internet and phone services for some buildings located on the capitol mall are temporarily unavailable. This issue is affecting certain state agencies and is limited to offices located on the capitol mall and some satellite offices.
The outage has also caused a disruption to Arizona Department of Transportation MVD services, including MVD and Authorized Third Party offices, as well as ServiceArizona.com. Customers are urged to delay office and online visits until this problem is corrected. Certain customer facing payment portals may also be affected.
Multiple vendors are on site and working hard to restore services as soon as possible.
Updates will continue to be provided.
Thank you for your patience during this disruption.
Agencies Impacted
Dept of Administration
Dept of Revenue
Dept of Agriculture
Industrial Commission of Arizona
Corporation Commission
Dept of Juvenile Corrections
Dept of Economic Security
Dept of Transportation
Dept of Health Services
Dept of Real Estate
Dept of Insurance
Commission for Deaf and Hard of Hearing
Update - 9.33 AM - Carrier technicians are working on isolating the fiber cable damage. AZNET and Fishel Technicians are on the Capitol Mall to investigate a temporary solution.
We are tracking the issue through Remedy ticket 226156.
CenturyLink is currently experiencing a Voice and Network intermittent outage.
We have notified the appropriate groups and are looking to provide a solution as soon as possible. We apologize for the inconvenience.
We are tracking the issue through Remedy ticket 226156.
Today Oct 31st we received reports of errors when attempting to state related Google services. We have notifified the appropriate groups and are currently awaiting a resolution.
This is currently being tracked with ticket '190539' in Cherwell.
Today,August 19, 2019, 1:30am, we started receiving reports of issues regarding Gmail, calendar and drive are returning an error 500.
We are working on identifying what is causing the issue. We have a master ticket 172609 open at ADOA.
At this time, we do not have and ETA. For up to date information, you can visit our AZCOOP.gov website. We apologize for this inconvenience, and hope to be able to report a resolution to this issue shortly.
Today August 7th, we started receiving reports around 9am that Talent Management is not reachable.
This is a known issue and we are working to provide a solution as soon as possible.
As of 10:30am, this is still an on-going issue.
11:45am, Issue is resolved. Access to Talent Management can resume as expected.
The problem with Google Calendar should be resolved. We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better.
There is a GOOGLE base Calendar issue that we are in process of resolving with the vendor. There is no ETA at the moment.
We have informed the appropriate groups and are looking to provide a solution as soon as possible.
This evening, the Palo Alto Next Generation Firewall implementation into the Oregon transit path of the ADOA AWS environment was completed successfully at approximately 9:10 PM.
In order to accomplish this successful implementation, the route limits on AWS were increased from 200 to 500. This route limit increase will allow for future scalability as necessary.
All testers on the Tester's Bridge Line reported no issues and/or outages as a result of this change event.
This will be the last posting on AZCOOP for this change event.
It's 9:00 PM, and all testers who have been on the Testers Bridge have tested successfully.
This includes testers from: Dept. of Insurance; ADOA-ASET Cloud Operations; ADOA-ASET Data Integration, and AFIS.
Portal has decided not to test this evening, due to no outages or issues since the changeover.
We are waiting to hear from AESP - an email was sent to the testers from this team.
The next posting will be published at 9:30 PM.
At 8:30 PM, an issue was experienced with a route limit on AWS. A trouble ticket was immediately sent to AWS to increase the route limit from 200 to 500.
Testers have been given the GO to test, and so far, all testers are reporting that everything is looking good.
The next posting will be published at 9:00 PM.
The time is now 8:00 PM, and we are starting the change event for the Palo Alto Next Generation Firewall Implementation into the Oregon path of the ADOA AWS environment.