Lumos game analytics

Discussion in 'Assets and Asset Store' started by Matthew Miner, May 27, 2011.

  1. Brad Keys

    Brad Keys

    Member

    Joined:
    May 1, 2006
    Messages:
    146
    You bet it does. We've done extensive testing with web, iOS, PC, and Mac standalone.
     
  2. sonicviz

    sonicviz

    Member

    Joined:
    May 19, 2009
    Messages:
    852
    A1. ty!
     
  3. homeros

    homeros

    Member

    Joined:
    Dec 23, 2009
    Messages:
    112
    I'm getting this error when I import your package:

    Assets/Editor/Lumos/LumosMenus.cs(40,22): error CS0117: `Help' does not contain a definition for `BrowseURL'

    I think another asset I got from the asset store also has Help class and that one is causing this issue. Is there any workaround for this?
     
  4. Matthew Miner

    Matthew Miner

    New Member

    Joined:
    Aug 29, 2005
    Messages:
    319
    This error can be resolved by modifying line 40 of LumosMenus.cs to read UnityEditor.Help.BrowseURL(documentationUrl); (that is, add the UnityEditor namespace before the Help class).

    The Help class comes built-in to Unity, so it's a bad idea for a third party add-on to include its own. Do you know which Asset Store package is doing this?
     
  5. homeros

    homeros

    Member

    Joined:
    Dec 23, 2009
    Messages:
    112
    Thanks for the help, it's Advanced Particle Effects. I've deleted it to try your package and I really liked it. Though I have one more question.

    I've set up Lumos package and used one Debug.Log (i've enabled logs from lumos object) and one Lumos.Event call. Then I started and stopped game a few times. Only two of these showed in both live feed and player count (with correct software, hardware etc..) but the logs and events didn't show up. Are these updated daily or am I doing something wrong?
     
  6. Matthew Miner

    Matthew Miner

    New Member

    Joined:
    Aug 29, 2005
    Messages:
    319
    Events and logs are queued and sent on a timer rather than being pushed to the server immediately. This means that if you stop the game, any unsent logs and events will be lost. The default timer length is 30 seconds. In practice we've found this generally isn't an issue, but obviously it can be confusing when testing.
     
  7. homeros

    homeros

    Member

    Joined:
    Dec 23, 2009
    Messages:
    112
    I wasn't confused about that. It was pretty clearly stated in docs. But for some reason 3 of the 5 tries I've played in editor didn't show up in player count and none of the logs and events showed up (it's been a few hours since i tried).

    Two tries that showed was the first two and had some more time between them (like a few mins). Maybe that's the reason but I still couldn't figure exactly how these analytics work. Do you keep track of ip and not count the same player twice? Was the problem something on my hand?
     
  8. Matthew Miner

    Matthew Miner

    New Member

    Joined:
    Aug 29, 2005
    Messages:
    319
    Ah sorry, I misunderstood the problem you're having. Player count will only go up once for each machine the game is played on. It keeps track of this with an entry in PlayerPrefs. Logs and events should record regardless though; it sounds like there might be an error happening somewhere on our servers. We'll look into the issue. Event recording has been a bit flaky lately, so it could be related to that.
     
  9. homeros

    homeros

    Member

    Joined:
    Dec 23, 2009
    Messages:
    112
    Did you have a chance to look at this? I've tried events again today and it didn't work. And I've noticed one more thing, when I enable "Record Log" on Lumos object, it gets disabled when I start the game.
     
  10. Brad Keys

    Brad Keys

    Member

    Joined:
    May 1, 2006
    Messages:
    146
    I'm getting the same issue. A temporary workaround is to change the settings directly in the prefab. Make sure to put your secret key in as well.
     
  11. homeros

    homeros

    Member

    Joined:
    Dec 23, 2009
    Messages:
    112
    Changing the prefab did the trick, though I still can't use events and logs :/ But in the end I really liked the simple layout of your admin panel and ease of use in unity class. What are your plans for "New (Awesome) Features" in premium membership :) ?
     
  12. Brad Keys

    Brad Keys

    Member

    Joined:
    May 1, 2006
    Messages:
    146
    We have been working away for a while now at something we really think the Unity community is going to be excited about. But! We're keeping it a secret until we get a bit closer to launching. Expect much more of the same simplicity and ease of use.
     
    Last edited: Mar 29, 2012
  13. virror

    virror

    Member

    Joined:
    Feb 3, 2012
    Messages:
    1,863
    Any way of exporting all of the statistics for one game to a single file?
    Or at least to be able to export everything and it ends up in a folder with everything exported separately, would also work.
     
  14. JTown.

    JTown.

    Member

    Joined:
    Jan 9, 2010
    Messages:
    863
    Hey I have Lumos integrated in an iOS app I'm working on but have yet to really set up all the analytics I need. Wanted to make sure of a few things because the deadline is tight and I want to mitigate risk.

    -Have apps using Lumos had any trouble getting approved on the app store? Are there apps on the store using it?
    -Are there any rules I should follow for gathering analytics? I know Apple is touchy about deviceInfo after some leaks.

    An example of that second one -- I want to track user framerate in a way that identifies the device. Like avgFramerate_iPad2 and avgFramerate_iPad3 -- any insight on whether or not that'd be a problem?

    I guess if I needed to, I could try to do it on an existing app I have and see if the update gets through.
     
  15. Matthew Miner

    Matthew Miner

    New Member

    Joined:
    Aug 29, 2005
    Messages:
    319
    Not yet, no. You can export all the logs into a CSV file, but not all the stats for the game. It's a feature we plan to add, but I can't promise a date.

    No games with Lumos installed have been rejected by Apple, nor do we believe there would be any reason to. There are indeed games on the App Store using Lumos. We turn off gathering system info (operating system, RAM, etc.) on iOS to comply with Apple's terms of service and we don't use the UDID for anything. It's hard to say how strict they are on gathering device info -- it's understandable why they might want to prevent the operating system from being reported, but it seems unlikely that having numbers on the device itself would be cause for concern. We encourage consulting Apple itself on this matter.
     
  16. virror

    virror

    Member

    Joined:
    Feb 3, 2012
    Messages:
    1,863
    Any way to see further back in the "New Players" graph? How can i export the whole graph, or at least all the values from it? If i do the image export i only get the current view. I think its very important to be able to export all of that data and not just the current window.
     
  17. Matthew Miner

    Matthew Miner

    New Member

    Joined:
    Aug 29, 2005
    Messages:
    319
    Not yet. We'll add this in the future, as well as the ability to download a CSV of all historical data, though I can't give a definite timeline of when this will be available.
     
  18. virror

    virror

    Member

    Joined:
    Feb 3, 2012
    Messages:
    1,863
    Ok, but the old data is saved right? Just not possible to see it yet?
     
  19. Matthew Miner

    Matthew Miner

    New Member

    Joined:
    Aug 29, 2005
    Messages:
    319
    Correct. We just haven't built a mechanism yet for retrieving and displaying data older than a month.
     
  20. Brad Keys

    Brad Keys

    Member

    Joined:
    May 1, 2006
    Messages:
    146
    The issue with Events has been resolved. Hurrah!
     
  21. JTown.

    JTown.

    Member

    Joined:
    Jan 9, 2010
    Messages:
    863
    Thanks. Do you happen to know if Apple's default app EULA covers typical analytics? Seems to be enough from what I can tell,

    But I'm no lawyer. Guess I'm just curious if developers typically have to have a custom EULA when using analytics or not.

    Apple's standard EULA: http://www.apple.com/legal/itunes/appstore/dev/stdeula/
     
  22. Matthew Miner

    Matthew Miner

    New Member

    Joined:
    Aug 29, 2005
    Messages:
    319
    I can't recall any games, using Lumos or otherwise, that present a custom EULA to users. Apple's standard EULA appears compatible with the information that we collect -- no personally identifiable data, and all for the purpose of identifying problems and improving your product. I'm confident that nothing we record conflicts with what the user has already agreed to, so I think you're fine.
     
  23. homeros

    homeros

    Member

    Joined:
    Dec 23, 2009
    Messages:
    112
    Nice, I missed this post last week. I'll try events again, thanks.
     
  24. homeros

    homeros

    Member

    Joined:
    Dec 23, 2009
    Messages:
    112
  25. Matthew Miner

    Matthew Miner

    New Member

    Joined:
    Aug 29, 2005
    Messages:
    319
    Just let us know if you run into any further difficulties. From our testing everything appears to work fine, but we're always on the lookout for edge cases we may have missed.
     
  26. JTown.

    JTown.

    Member

    Joined:
    Jan 9, 2010
    Messages:
    863
    Today I'm getting:
    "You are trying to load data from a www stream which had the following error when downloading.
    500 Internal Server Error" in the editor

    From Lumos.Log("Response: " + www.text); in SendCoroutine in LumosWWW

    edit: it also seems to cause crash (lldb) when running on device from xcode - disabling lumos script stops the crash
     
    Last edited: May 9, 2012
  27. Matthew Miner

    Matthew Miner

    New Member

    Joined:
    Aug 29, 2005
    Messages:
    319
    Thanks for letting us know, looks like there were a few glitches in back-end changes we deployed last night. It should be fixed now.
     
  28. virror

    virror

    Member

    Joined:
    Feb 3, 2012
    Messages:
    1,863
    Any date when the next update is coming out?
    Looking forward to the ability either exporting all data or at least be able to access the data that's old and outside of the graphs : )
     
  29. Matthew Miner

    Matthew Miner

    New Member

    Joined:
    Aug 29, 2005
    Messages:
    319
    Unfortunately we can't promise release dates for new features at this time (not terribly useful for you I know, but we'd probably be wrong about any estimates anyway). We'll definitely make it known when new stuff ships.
     
  30. virror

    virror

    Member

    Joined:
    Feb 3, 2012
    Messages:
    1,863
    Still pretty far away then i guess : p
    Thanx for the answer anyway, great piece of software : )
     
  31. Luke.Houlihan

    Luke.Houlihan

    Member

    Joined:
    Jun 26, 2007
    Messages:
    193
    I'm getting an error from Lumos,

    "You are trying to load data from a www stram which had the following error when downloading.
    400 bad request"
     
  32. Matthew Miner

    Matthew Miner

    New Member

    Joined:
    Aug 29, 2005
    Messages:
    319
    Events and logs are currently down for maintenance, which is where the 400 error is coming from. We hope to have everything operational soon.
     
  33. Luke.Houlihan

    Luke.Houlihan

    Member

    Joined:
    Jun 26, 2007
    Messages:
    193
    Bummer, I had to remove it from the ios project because the error was causing a lock-up when run from xcode.
     
  34. Matthew Miner

    Matthew Miner

    New Member

    Joined:
    Aug 29, 2005
    Messages:
    319
    That's odd (and unexpected) that it would cause a lockup. The error should be handled by Unity like any other. Thanks for letting us know though, maybe I can find a way to prevent that from happening.
     
  35. seanlloydbooth

    seanlloydbooth

    Member

    Joined:
    Jan 12, 2012
    Messages:
    59
    Also getting the same problem. Is there a quick fix? XCode doesn't say anythings wrong when it's running, it just freezes and pauses the build.

    EDIT: Found what was freezing XCode - In LumosWWW.cs, Line 99:

    Code (csharp):
    1.  
    2. // Display returned info if there is any
    3.             if (response.Count != 0  response.Contains("result")) {
    4.                 var result = response["result"];
    5.                 Lumos.Log("Success: " + result);
    6.             }
    7.  
     
    Last edited: Jul 6, 2012
  36. Matthew Miner

    Matthew Miner

    New Member

    Joined:
    Aug 29, 2005
    Messages:
    319
    This is good to know. I would have expected the try/catch statement in LumosWWW to handle any errors properly, but perhaps XCode does something extra with exceptions. Until we release an update I recommended commenting out or deleting lines 97 to 103 in LumosWWW.cs. They're only useful when debugging anyway, so removing them won't have any negative side effects.
     
  37. Matthew Miner

    Matthew Miner

    New Member

    Joined:
    Aug 29, 2005
    Messages:
    319
    We updated the Lumos Unity package to prevent these errors from occurring. We highly recommend that everyone install the update; it will also help us manage traffic spikes in the future. It's available on the Asset Store or on our downloads page.
     
  38. sonicviz

    sonicviz

    Member

    Joined:
    May 19, 2009
    Messages:
    852
  39. Matthew Miner

    Matthew Miner

    New Member

    Joined:
    Aug 29, 2005
    Messages:
    319
    Information sent to Lumos is not encrypted, no. Should we record more sensitive information in the future the extra overhead will be worthwhile, but it's not a priority with the kind of data we record.
     
  40. sonicviz

    sonicviz

    Member

    Joined:
    May 19, 2009
    Messages:
    852
    Thanks for the confirmation.

    Doing some testing now. Nice and easy to implement!
    Could you clarify what unit the numbers are on the website for the default recorded events:
    Recorded


    Hits vs Values? I'm assuming hits are a unit figure of number of times level is started(?), but what are the values (Avg/min/max/sum) in? Are these in minutes?
    Some more info on the axis would be great.
    Also not sure how to interpret the "Compare With Total Players" % figure. How is that being calculated?


    ty!
     
  41. Matthew Miner

    Matthew Miner

    New Member

    Joined:
    Aug 29, 2005
    Messages:
    319
    The events system could certainly be clearer. The units for Level Completion Time is in seconds. So, a value of 3600 means that the level was completed in 60 minutes. The Level Started event is simply a count of the number of players who have started a level. This is only recorded once for each player. That is, if 300 players in total make it to level three, that's the number that the event shows. When you choose the option to compare with the total percent of players, it displays the event count divided by the total number of players recorded. So again, if 300 players make it to level three but 1000 players have played the game in total, it will display 30%. With this statistic you might see, for example, that only 10% of players have made it past a particularly tricky level.

    For arbitrary events that you add, the units are in whatever you decide. If you're recording the time that players take to complete a lap in a racing game, it will be a time in seconds. The average is the mean time that all players get, the minimum the fastest time, and the maximum the slowest. The sum will only be useful for some types of events. A game like Angry Birds might be interested to know how many birds have been shot overall in level five, for curiosity's sake. In this case the sum would show this value. It's a flexible system that lets you find and compare many stats, though admittedly its interface for doing so could be improved.

    We'd love to learn more about what people are interested in using events for so that we can improve the system. We've had a lot of great suggestions sent to us, but I'm sure there are tons of use cases we haven't yet considered.
     
  42. sonicviz

    sonicviz

    Member

    Joined:
    May 19, 2009
    Messages:
    852
    Thanks for the info.
    Is that level started count unique per installed version, unique per play? I guess it's unique per play, as installed would need another LOD?

    ty!
     
  43. sonicviz

    sonicviz

    Member

    Joined:
    May 19, 2009
    Messages:
    852
    Another Q:

    What's the standard or best practice for notifying the user that you are collecting anonymous stats with Lumos?
    Do you say nothing?
    Post on website?
    Offer a dialog in game that asks for permission and sets an option on/off?
    or ?

    ty!
     
  44. Matthew Miner

    Matthew Miner

    New Member

    Joined:
    Aug 29, 2005
    Messages:
    319
    It's unique per machine (which I believe is what you're describing). That is, a flag is set in PlayerPrefs signifying that the event has been recorded and should never be recorded again, which persists between plays.

    The more transparency, the better. While we don't collect any personal information about a player, I recommend having an option somewhere in the game to turn Lumos off. Noting on your website that you use analytics to collect gameplay and diagnostics information isn't a bad idea either. We've had several requests for a way to disable Lumos other than deactivating the game object, so in the next update to the Unity client (likely in a couple of days) we'll have this option available.
     
  45. goodhustle

    goodhustle

    New Member

    Joined:
    Jun 4, 2009
    Messages:
    310
    Hi Matthew,

    Lumos has been extremely easy to integrate so far, and you and your team have been great about responding to feedback and answering questions in a timely manner. Kudos where kudos are due!

    My impression so far of Lumos is that it's an excellent system for doing pre-release analytics and beta testing and tuning.

    For production use, there are a couple limitations of the Events and core system that I would love to hear your thoughts about:

    1) Only the aggregate overall stats are shown in a chart over time. In order to determine whether "live" tuning changes are having an effect, the Events system would need to allow for chart plotting over time. Events are only currently tabulated over all time.

    2) If one wanted to track each game version's events separately, there's nothing built-in at the moment. The hacky method would be to prefix a game binary version to all Event keys when reporting to a server, so I guess there's that. Do you have any plans to allow for versioning of event data?

    One of the key use cases of event analytics in production is to determine whether changes are having a desired effect, so the ability to determine this is hampered if one cannot separate data by date range for "live" updated games such as webplayers or for distributed updates such as standalone builds.

    I'm looking forward to a disable flag for the Lumos client! It would be great if this did not disable the feedback, as a user that does not want to participate in anonymous stat tracking might very well like to use a feedback form. Thanks for your hard work!

    -Gordon
     
  46. Matthew Miner

    Matthew Miner

    New Member

    Joined:
    Aug 29, 2005
    Messages:
    319
    Both of your proposed changes to the events system would definitely be useful. Currently we only keep summarized event data (i.e. we throw away individual events after they've been factored into the mins/averages/etc.), so supporting date ranges and plotting changes over time would require a rewrite of the system. It's due for an upgrade though, and we've been listening closely to what developers are requesting in preparation for one. We'd love to hear more use cases and what sort of information developers are tracking / want to track.

    As for versioning events, you could create a new game on Lumos for each version and supply the game with different secret keys accordingly. It's a hacky solution and not even close to being user friendly, but it might give you a bit more insight into how players react to changes.
     
  47. Default117

    Default117

    New Member

    Joined:
    Mar 13, 2011
    Messages:
    131
    +1 for being able to track per version. Ideally it would be best to keep it under a single game tab as to quickly bring up side by side graphs, or layered graphs for easy comparison between version releases.
     
  48. Nition

    Nition

    Member

    Joined:
    Jul 4, 2012
    Messages:
    326
    I've just put Lumos into my game-in-development and it integrated very easily. Thanks for making a great free tool. I'd definitely find it helpful to be able to view individual data points. Avarages and outliers don't always give the whole story.

    Edit: A couple more things:
    - Deleting a game doesn't seem to work. It's giving me a 500 Internal Server Error in both Chrome and IE.
    - Is there a way to delete a player or clear the events? A had logging from the editor on and accidentally got a ridiculous impossible score, which has kind of stuffed up my data.
     
    Last edited: Aug 30, 2012
  49. Nicolaj Schweitz

    Nicolaj Schweitz

    Member

    Joined:
    Feb 25, 2009
    Messages:
    29
    I love this tool!
    It really suits Unity with its simplicity.

    I am not sure I understand how to get the debug logs recorded.
    It seems that logging is not recorded from Awake() and Start().
    It seems to work fine when fired from Update() and OnGUI().

    Code (csharp):
    1.  
    2. using UnityEngine;
    3. using System.Collections;
    4.  
    5. public class LumosButtons : MonoBehaviour {
    6.    
    7.     private bool timerIsRunning = true;
    8.     private float startTime;
    9.    
    10.     // this is not sent
    11.     void Awake () {
    12.         Debug.Log ("log info from awake");
    13.         Debug.LogError ("log error from awake");
    14.         Debug.LogWarning ("log warning from awake");
    15.     }
    16.    
    17.     // this is not sent
    18.     void Start () {
    19.         Debug.Log ("log info from start");
    20.     }
    21.    
    22.     // this is sent
    23.     void Update () {
    24.         Debug.Log ("log info in update");
    25.     }
    26.  
    27.     // this is sent
    28.     void OnGUI () {
    29.         if (GUILayout.Button ("send log info")) {
    30.             Debug.Log ("log info from OnGUI");
    31.         }
    32.     }
    33. }
    34.  
     
  50. Matthew Miner

    Matthew Miner

    New Member

    Joined:
    Aug 29, 2005
    Messages:
    319
    Agreed. Keeping individual data points has been requested a lot lately, and we're currently working out how to store all this data while keeping costs reasonable (you could use a service like MixPanel, but for most games the prices would be astronomical).

    Ah, so it does. We'll see about getting that fixed.

    No currently, no. It's been on our todo list, but the ability to delete data hasn't been implemented yet. We'll work on that.

    We can't guarantee that recording logs will be possible in Awake or Start. Our own script's Awake function registers the callback function that Unity sends logs to, and depending on the script execution order this may happen after your own Awake function. Unfortunately I'm unsure of how to remedy this. We'll certainly look into it and experiment, but I have a hunch that it might be a limitation we're unable to work around.