Something broke again.
For the summary of the break:
|
|
Redis (faster reliable logins) |
Database data up to date |
No more session timeouts (UVLe usable) |
| 24-Aug |
8pm |
Yes |
No |
Yes |
| 25-Aug |
9am |
No |
Yes |
No |
| 25-Aug |
8pm |
No |
No |
Yes |
| 26-Aug |
1pm |
No |
Yes |
Yes |
For those that want to learn more or are curious about devops:
The scheduled downtime introduced a new problem that we haven't encountered before in UVLe as the administrators of the platform, we couldn't log in. Since UVLe is encrypted especially in user sessions and login credentials, we couldn't do much during the first day that it happened. We essentially we're debugging with half the tools that we could use before. So that was Friday. We tried multiple ways to solve the issue, but none of them worked. (
https://docs.moodle.org/500/en/Session_handling)
Saturday, we essentially read through forums and fixes to try to figure out what was wrong with our system. This was when we took down access to UVLe by restricting network access to limited use. Take note that everything is backedup before any action is taken, and if the action doesn't work, we revert to a previous backup to ensure that the error doesn't pile up. This takes a lot of time.
Sunday was when a breakthrough happened, one of the devs were able to login after following in one of the forums (forgot the link) and from there, he worked his way to changing the session handling. However, the data that was retained was from Aug 15, everything after was removed. Sunday, night, we tried recovering one of the backup databases to no success.
Monday morning, a backup of the database last Thursday was loaded and it had the correct data, up to Aug 21. However, the login issue persisted with this database (telling us that we only knew half of the issue and only solved that half). Monday afternoon, starting from Sunday night's database, a database recovery was conducted. However, the recovery from the primary database still resulted in data from August 15. Monday night, changes were done to the database cluster to ensure that the correct data will be restored.
The restoration finished 1pm today (as the restoration died sometime around 2am and was only restarted at 6am). And this is the current UVLe. We'll be launching a new one with better updates and UI sometime this semester. Thank you for your patience and we apologize for the inconvenience.
Best,
ILCD
