Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Linux headless server cpu usage growing

Discussion in 'Multiplayer' started by cpasjuste, Sep 2, 2013.

  1. cpasjuste

    cpasjuste

    Joined:
    Apr 2, 2012
    Posts:
    154
    Hi,

    I'm wondering if anyone played with the new headless mode provided by Unity 4.2 on Linux. My problem is the CPU usage is constantly growing up over days, even if no connection is made to the server.

    When I start the server it is eating around 5% CPU usage (top) but it constantly grow over days, for example after 10 days its now eating around 65% of CPU usage.

    Did anyone experienced this sort of problem, or have some advices about what could be the cause ?

    Many thanks in advance,
    David.
     
  2. akyoto_official

    akyoto_official

    Joined:
    Feb 24, 2013
    Posts:
    52
    Yes, I'm having the same problem as shown here: http://forum.muchdifferent.com/unit...cussion/1518/sudden-change-in-cpu-consumption

    It is slowly, but constantly growing.


    This is a single lobby application that listens for peers connecting to it. It started at 1% CPU consumption and it has been running for 2 days now. After it reached 30% I restarted it and it was at 1%, again.
    It currently makes it impossible to let a server run with 100% availability and at high CPU usage it can and will reduce the response time of the applications that really need the CPU.
     
  3. cpasjuste

    cpasjuste

    Joined:
    Apr 2, 2012
    Posts:
    154
    Great im not alone. I guess we should open a bug report, I'll try to do that if I find the time.
     
  4. any_user

    any_user

    Joined:
    Oct 19, 2008
    Posts:
    374
    We're also experiencing that problem on our servers.. it's pretty annoying because it's very hard to find the reason for it. In our case it happens after a few hours when no player is connecting.
     
  5. FuzzyShan

    FuzzyShan

    Joined:
    Jul 30, 2012
    Posts:
    182
    I am also having the same issue, is everyone here using uLink as server itself? Does this happen on windows?
     
  6. Glader

    Glader

    Joined:
    Aug 19, 2013
    Posts:
    456
    Could it internally be slowly collecting too many objects, not releasing them when they should, and somehow be iterating over them when they shouldn't be?

    Does the RAM usage increase ever so slowly?
     
  7. FuzzyShan

    FuzzyShan

    Joined:
    Jul 30, 2012
    Posts:
    182
    Last edited: Jan 8, 2014
  8. any_user

    any_user

    Joined:
    Oct 19, 2008
    Posts:
    374
    is there any news on this? any ideas for a workaround?
     
  9. FuzzyShan

    FuzzyShan

    Joined:
    Jul 30, 2012
    Posts:
    182
    sent them a message, waiting for a reply back.
     
  10. jjobby

    jjobby

    Joined:
    Nov 28, 2009
    Posts:
    160
    It seems the issue won't be fixed in the near future. Do you think that changing to Window is the best decision?
     
  11. FuzzyShan

    FuzzyShan

    Joined:
    Jul 30, 2012
    Posts:
    182
    seems like that will be the only option left, I find it to be pretty harsh on how unity takes importance over their instance to be able to run ok over linux, I also think that linux servers are more important compare to actual linux single player games.

    Anyways, just to let other people know that if your sever is unity instance based, then you should think twice about it.
     
    Last edited: Feb 9, 2014
  12. rsoares

    rsoares

    Joined:
    Mar 19, 2013
    Posts:
    35
    Did you guys tried to limit the FPS of the build ?
     
  13. FuzzyShan

    FuzzyShan

    Joined:
    Jul 30, 2012
    Posts:
    182
  14. FissionFlame

    FissionFlame

    Joined:
    Sep 9, 2013
    Posts:
    6
    I have 3 ideas for you to try out

    1. Lets say you scene is called TownA, when the server reaches 0 players connected, change TownA to an idle scene with nothing on it.
    When a player attempts to connect, make it switch back to TownA or any previous map.

    2. Try initializing the map its sitting idle on. like changing the map to itself. If the CPU usage goes down, implement some code that initializes your map every so often while players aren't connected.

    3. If it goes away after restarting the unity server program, then do something like this
    At 1 am, check if players are connected, if so, check again every 5 minutes till no players are connected.
    Once server is empty, restart and wait till 1 am the following night.
     
  15. GooseSerbus

    GooseSerbus

    Joined:
    Jun 19, 2013
    Posts:
    4
    I've been looking at a similar issue for a while now - we've noticed the 'fixed' framerate of our servers has been increasing over time, which may be the same as this issue, as an increased framerate would likely cause an increase in CPU usage that you've seen.

    I built a test application which runs as fast as possible (uncapped framerate, no graphics etc.) and prints out the current frame number once a minute. I've been running this for days at a time to monitor what's been happening and what I've noticed on Linux over multiple tests (with both headless and windowed) is that there are distinct 'jumps' where the average framerate over the last minute has increased by several thousand FPS and remained there - this only continues over time and after several days you can reach 80k FPS having originally started out around 1k.

    An example graph:

    $muap1t.png

    (The scale on the side is 0-14,000 FPS and the test was only running for about 15 hours)

    In our servers with a fixed framerate I assume what we're seeing is the first jump after several days and leaving them running for a long time will eventually result in further jumps and increased framerate/CPU usage. I thought this issue might have been related to a bugfix in 4.2 related to negative timescales, but that doesn't seem to be the case (the timescale stays the same for the duration of the test). I also suspected that it could be related to the frame number overflowing after 2^31-1 frames (it's an int), but this doesn't seem to affect it (the jumps happen well before the overflow and it will overflow several times without issue).

    I've tested this in 4.1, 4.2 and 4.3 and they all exhibit the same behaviour, although sometimes windowed mode didn't have the issue and the framerate remained fairly stable. I can only assume that this bug has been around since Linux publishing was added in 4.0. Testing on a VM or hardware gave the same result, for reference I've been testing on Ubuntu 12.04.4 LTS (Precise Pangolin) and Debian 7 ("wheezy"). I have not seen this behaviour in a standalone Windows build (I was planning on trying OSX but haven't done so yet).

    The only 'workaround' I can think of is sleeping the main thread in C# for a 'best guess' amount of time to make the frames last longer after the jump, the engine will be sleeping to fix the framerate so it's pretty similar. We've not yet tested this though.

    Anyway, I am going to be submitting a support request to Unity.
     
  16. cpasjuste

    cpasjuste

    Joined:
    Apr 2, 2012
    Posts:
    154
    Well I think you did a really good (and probably long) job here. Submitting a support request with those informations will probably help a lot.

    I may also have the time to come back on it too soon. I do have a very nice counter-strike like game for mobile in a pretty good shape since maybe a year now, but this sort of game NEED a working Linux server so I had to stop the development :/

    Thanks GooseSerbus.
     
  17. GooseSerbus

    GooseSerbus

    Joined:
    Jun 19, 2013
    Posts:
    4
    Reproduced and confirmed as an bug by Unity who have passed it onto their Linux dev team - they saw both the framerate jump and CPU usage increase. I was not given any idea of when it might be fixed, probably partly because they don't know what's causing it yet.

    They said my suggested workaround of capping FPS in script should work and they also suggested using the Time.realtimeSinceStartup property to see how long how passed since the last frame and doing the Sleep call after a yield to WaitForEndOfFrame(); I'm probably gonna do this at some point over the weekend and test over the coming week to see if it causes any problems.
     
    Last edited: Apr 4, 2014
  18. FuzzyShan

    FuzzyShan

    Joined:
    Jul 30, 2012
    Posts:
    182
    I end up switching back to Windows server for Unity, my process use to grow up to 45% cpu usage after couple days, but now it stays at 0% on windows.

    I am assuming that this has to do with mono .net compatible issue against linux, I think all Unity based servers are pretty much has to be run on windows, drop linux for game servers, you can still put database stuff on linux.
    This won't be an easy issue to fix... Hopefully it saves a lot of you guys out there sometime...
     
    Last edited: Apr 8, 2014
  19. NaXDy

    NaXDy

    Joined:
    May 12, 2014
    Posts:
    14
    I'm planning on running a headless server on Linux, but reading the above, I might not want to do that.

    Has anyone tried running a headless windows unity server on Linux using Wine? If so, does that also increase the CPU load over time or is it relatively safe?
     
  20. rsoares

    rsoares

    Joined:
    Mar 19, 2013
    Posts:
    35
    Wine itself uses CPU power, so migth aswell stick with windows.
     
  21. Yukichu

    Yukichu

    Joined:
    Apr 2, 2013
    Posts:
    420
    Any news on this or the workaround?

    So basically I'd start a coroutine that will:
    takes the time
    yield return new WaitForEndOfFrame()
    sleep the main thread for... an approximation of what should be the frame rate time - stored time

    Annoying.
     
  22. GooseSerbus

    GooseSerbus

    Joined:
    Jun 19, 2013
    Posts:
    4
    I've been running my workaround for a few weeks now and it seems pretty stable, however it does look like they've fixed this in 4.5:

    http://unity3d.com/unity/whats-new/unity-4.5

    If you're stuck pre-4.5 (I am for now) then my workaround was something like the following which worked fine on Windows and Ubuntu, but I had some problems on Debian, YMMV. I do not consider this code tested so please do your own testing before using.

    Code (csharp):
    1.  
    2. using UnityEngine;
    3. using System;
    4. using System.Collections;
    5. using System.Threading;
    6.  
    7. // Fix the framerate of your application instead of using Unity's built-in version
    8. public class FixFramerate : MonoBehaviour {
    9.    
    10.     private double TargetFramerate = 20.0; // fps (default)
    11.     private double TargetDeltaTime;
    12.     private double LastSyncEnd = 0.0;
    13.  
    14.     public double GetTargetFramerate() {
    15.         return TargetFramerate;
    16.     }
    17.  
    18.     public void SetTargetFramerate(double _targetFramerate) {
    19.         if (_targetFramerate <= 0) throw new ArgumentException("Target framerate must be positive", "_targetFramerate");
    20.         TargetFramerate = _targetFramerate;
    21.         // We wait a bit less (0.1ms) that the true target delta to accomodate for small fluctuations
    22.         TargetDeltaTime = (1.0 / TargetFramerate) - 0.0001;
    23.         Debug.Log("Framerate fixed to " + TargetFramerate + " fps");
    24.     }
    25.  
    26.     private void Awake() {
    27.         SetTargetFramerate(TargetFramerate);
    28.         // set Unity's target framerate to be uncapped  make sure vsync off (so we can manage it)
    29.         Application.runInBackground = true;
    30.         Application.targetFrameRate = -1;
    31.         QualitySettings.vSyncCount = 0;
    32.     }
    33.  
    34.     // We use a Coroutine so we can wait until the end of the frame
    35.     private void LateUpdate() {
    36.         StartCoroutine(SyncMainThread());
    37.     }
    38.  
    39.     private IEnumerator SyncMainThread() {
    40.         yield return new WaitForEndOfFrame();
    41.         // check nothing has changed the Unity target framerate
    42.         if (Application.targetFrameRate != -1) {
    43.             Debug.LogError("Application.targetFrameRate changed to: " + Application.targetFrameRate + " - resetting to -1 (uncapped).");
    44.             Application.targetFrameRate = -1;
    45.         }
    46.         double time = Time.realtimeSinceStartup;
    47.         // wait for enough time to pass for the requested framerate
    48.         double deltaTime = time - LastSyncEnd;
    49.         if (deltaTime < TargetDeltaTime) {
    50.             int sleepTimeMillis = Convert.ToInt32(Math.Floor((TargetDeltaTime - deltaTime) * 1000.0));
    51.             Thread.Sleep(sleepTimeMillis);
    52.         }
    53.         LastSyncEnd = Time.realtimeSinceStartup;
    54.     }
    55.  
    56. }
    57.  
     
  23. Patrick Juhl

    Patrick Juhl

    Joined:
    Jun 9, 2014
    Posts:
    1
    Indeed there this seems to be fixed in 4.5. Another alternative for uLink users is to use the TargetFrameRate fix available.

    Cheers,


    Patrick
     
  24. GooseSerbus

    GooseSerbus

    Joined:
    Jun 19, 2013
    Posts:
    4
    That fix is windows only, this is a Linux issue