Resolved - Slow logon and logoff on lecture theatre and open access computers

alert

[30 October 2015]

Update 30 October: We believe that the problems of slow logon and logoff to lecture theatre and open access computers are now resolved.  Following changes we made to the behaviour of the roaming user profile service, we have been monitoring the situation this week and the situation now seems stable.

Thank you for your patience while we investigated and worked on this issue.  We will be making further changes as a result of this to try to ensure that we do not see a recurrence of the problem, but if you see issues again please do report it to us through the service desk in the usual way.


Update 28 October: We believe we have made some progress in determining the cause of the issues.  Our current theory is that user profiles contain a significant number of files, in the tens of thousands. The impact of such high numbers mean that profiles apply very slowly at very high traffic times.

There are a number of tools which create such profile files, for example, web browsers such as Firefox and Google Chrome. We have identified particularly troublesome areas and excluded them from the roaming behaviour. We think that this is why things have been more stable this week. However, for those excluded folders, if there is functionality that relies on the data in them, then that functionality won't work as expected.


Update 23 October: There are have not been reports of slow logins today (Friday) in lecture theatres or student study areas. We did take some remedial action yesterday but reduced activity on a Friday makes impact difficult to judge. We are still exploring causes and have further action to increase capacity to tackle symptoms.


IT Services have been receiving reports of slow logins and logoff on lecture theatre and open access spaces over the past few days. We have been working to identify the problem but unfortunately the issue remains.

Please note that login or logoff will work but it may take some time. Attempting to log on to additional PCs certainly won't help and is likely to make things worse.  If you do encounter this issue please report it to the IT Service Desk, service-desk@bristol.ac.uk.

Cause

From investigating the logs on affected PCs at affected times, the problem seems to relate to profile syncing, which happens at log on and log off.

The profile syncing is happening at all times, it is just happening far too slowly. That is to say that at no time is the profile store unavailable.

We believe the problem is triggered by the spikes in logon/off activity around teaching. We think the problem is therefore being triggered in teaching space, but because student spaces share the same profile store they are being affected too.

However, we have not determined the root cause of the problem that is manifesting as slow profile syncing.

Please note

The problem is not with the local PC and there is no connection between this and Mediasite.  There is nothing installed on the PC related to Mediasite and no connection physical or otherwise between the PCs and the Mediasite recorders.

Actions

  • We are investigating the storage behind this profile store to see if there are issues there.
  • We have moved the presentation of this storage to a server which is less busy than the original one (unfortunately this does not seem to have helped a great deal).
  • We are investigating what workarounds we might be able to put in place if we cannot identify the root cause, that might improve the experience for our users.
  • We are continuing to investigate reported incidents to help identify a cause.