Off-site backup services for my Unity project

Hey guys,

I just wanted some ideas on good services/companies/websites that will automatically keep my Unity project folder backed up (off-site).

I'm asking this because I just tried "Carbonite" and I wanted to download a "backed up" project and Carbonite limited me to downloading 5000 files per download, and my project has 100,000+ files in it. So I had to manually go through and download 5000 files at a time, and try to keep them all in the same hierarchy etc..... It was a pain.

Long story short, I don't want to use Carbonite anymore and want to do something else. I'm looking at Dropbox right now (is that a good option?).

Something that allows me to have (or download) my entire project all at once would be nice. Any recommendations? Pro tips? Thanks

 

What about version control? Visual Studio Team Services is free for teams of 5, with unlimited repository sizes.

 

Thats the one I was trying to remember - thanks Tony.
How could I manipulate this to free up some space on my local drive filled with art/animation content. Hmmm?

 

Actual game content or source files?

 

Interesting, I'll have to take a look at that, thanks.

Actual game content included (if you're asking me).

 

If it were source, that would be easy, put it in a sibling of the assets folder, commit and delete locally. For game content about the only thing you can do is shift to asset bundles, using a second project, so the don't live in the project. But that isn't a trivial effort. Best option would be to just get a bigger drive.

 

Some DVD-Rs or an external HDD might be more convenient.

There's a bit of a learning curve to version control, but the first time it saves your project you won't begrudge the learning curve in the least. With plain cloud storage like Dropbox, if you've been saving a corrupt scene file for days, there's no easy way to get it back to a good version. But if you're using version control, you can just bring back an older version.

 

PlasticSCM is a source control system that specifically mentions being ideal for game development.

 

I use VSTS with both Git and TFS (depending on project). It's not always ideal for large or binary files but works well and it's free. I would highly recommend a source control system, but backup systems are also important and a separate consideration.

I use CrashPlan (www.crashplan.com) and have for years. It works really well for me... unlimited storage, pretty cheap and had faster upload times than Carbonite and one of the others I had tried. It also lets you build backup sets (different sets of folders for organization) and set some very advanced rules on retention, like how long to keep deleted files (up to forever), and versions of files so if something gets effed you can roll back to a previous version.

The website limits the number of downloads, but the desktop app lets you restore as much as you want to wherever you want. If I had one complaint though, the download times are a bit slow so if it's a massive project it could take some time to recover if you need to. But if you're using a source control system in conjunction, hopefully you want ever need to restore from the backup... it's a failsafe.

 

Just an update... CrashPlan announced they are ending personal backup services and concentrating on business offerings so my above recommendation only applies for business accounts.

 

I use this one. Seriously it is just awesome and reliable: https://www.arqbackup.com
Works with almost any Cloud, SFTP, NAS or local HDs. It works like Apples TimeMachine, just with more options. Which means you can revert to old versions too because it keeps a backup history with multiple versions.

Arq® automatically backs up all your Macs and Windows PCs.
Your files are stored securely, readable only by you.

Arq backs up your files securely and lets you keep control of your data. Arq is super simple to set up. It backs up your files to your own cloud account (Amazon Cloud Drive, AWS, Dropbox, Google Drive, Google Cloud Storage, OneDrive, and SFTP). Arq protects your data through strong encryption with a key only you know. Arq uses multiple threads and LZ4 compression for very fast throughput. Arq offers sophisticated file-exclusion filters, storage budgeting, extensive logging, email reports, many scheduling options, and full control over which networks and how much bandwidth Arq uses. Arq is availabe for Mac OS X and Windows.

True privacy and control
Arq encrypts your file data before it leaves your computer, so your files are never sent on the network in the clear. You choose the encryption password. We have no access to your files. All transmission over the network is through SSL/TLS.

Totally transparent
Your backups are stored in an open, documented format. We've published an open-source command-line utility so you can see for yourself your backups and directly restore your files from cloud storage. Other users have also contributed Arq-compatible utilities like arqinator.

Native apps
Both Arq for Mac and Arq for Windows are native apps built for maximum device fidelity and maximum performance. No Java means no Java-related upgrade headaches or performance problems.

Efficient
Arq does all it can to minimize uploads: Compression reduces upload sizes. Client-side de-duplication ensures Arq never uploads the same content twice. Rsync-style "rolling checksum" ensures only the changed parts of files are uploaded.

Unlimited in every way

• Arq backs up everything or just the files you choose.
• Arq doesn't skip videos or ignore certain file types
• Arq backs up files of any size -- 4GB, 40GB, it doesn't matter
• Arq backs up your external drives
• Arq backs up your network drives
• Arq doesn't delete backups of your external drives just because you haven't plugged them in lately
• Arq doesn't forcibly delete old backup records (unless you tell it to).
• Arq restores your files onto any computer.

 

$49.99... is that really a one time fee? That seems cheap.

Edit: Ahh I see... it requires you to purchase a separate storage account for your data. That would get really expensive for me, really fast, even if I used Glacier or Coldline.

 

Upgrading from one major release to another has a cost of $24.99 per user too.

https://store.arqbackup.com/

Alternatively you can pay $29.99 per user more at initial purchase for lifetime updates.

 

yuck. I haven't looked at many alternatives yet, but BackBlaze was recommended as a replacement. Still much more expensive than the family plan of crashplan if you have multiple PC's.

https://www.backblaze.com/cloud-bac...1QIVSbbACh3pig7GEAAYASAAEgLqWfD_BwE#crashplan

 

Lmao, a DVD. I'm laughing just imagining someone with their game source files on a DVD...

 

Those are 5 1/4" Floppy Disks.
Anyway an additional backup on optical discs is useful because it is EMP proof.
Something like this can happen any time: https://en.wikipedia.org/wiki/Solar_storm_of_1859

 

Seems like there would be a lot more serious problems than losing data at that point but wouldn't a flash drive work just as well as a dvd-r in an EMP situation?

 

No, in such an event a Flash drive (like any other electronics) would be destroyed.
Unless you put your computer and electronics perfectly isolated (no cables to the outside of the cage) into a faraday cage it would be destroyed.

 

That sucks, didn't know that. I never really thought about something like that happening.

 

You could store your important hardware in a primitive faraday cage when such a heavy solar storm ever hits earth.

 

Just backing up my project!
(Actually big places like Google use tape (modern tape though!) for backups because they're durable, low power consumption, and super high density.)

I recommended optical discs for archiving (not daily backups) because they have a longer shelf life than HDDs. They're less likely to get lost than loose thumb drives. And SSDs are nice but more expensive. But whatever works for you! I'm probably placing too much faith in cloud storage myself, but then again I'm paranoid I'm going to leave the oven on and burn down my house.

 

Bd-R/RW might be better for big source bases.

 

Don't mind me, just pushing a commit to the repository.

That bad boy can store up to 5MB of data!

 

Yeah, but what would the point of that be? We wouldn't even have functioning infrastructure to power it, let alone do anything with it. A solar storm of that magnitude would collapse society with the havoc it'd wreak.

 

Yep, there's no real point. If something of that magnitude happens, losing your personal data is the least of your worries.

 

They are aware of the problem and slowly trying to harden the power grid. http://www.washingtonexaminer.com/g...tric-grid-from-massive-attack/article/2620280
Such an event isn’t the end of the civilization. It would be just a huge outage over some months and a lot of people will die of starvation. As soon an everything is restored the world will be the same again, this is the moment when you wish you had an EMP-proof backup.

 

Yeah, uh... no. It's not just the power grid. It's literally all computer-connected infrastructure as well, also manufacturing, communication, transportation logistics, finances, entire economic systems really, all sorts of things. We are dramatically underprepared for this sort of thing and backing up your Unity projects is going to be ultimately pointless.

 

Of course it is all this stuff. And all this can be rebuild comparable quickly. It’s not much different to a destroyed country after a full scale war.

 

On the contrary it's considerably different. One destroyed country can quickly rebuild because it's only a single country out of many that have at least the same levels of capability. Recovering a single country can be achieved quickly because you can bring in fully manufactured goods from other countries.

A solar flare of the magnitude you're describing is a completely different situation because it won't simply destroy one country. It would destroy every country reliant on technology capable of being affected by EMP.

You wouldn't be able to rely on anyone to supply the manufactured goods for rebuilding because the countries that would have normally provided them would no longer have the ability to manufacture them let alone transport them to where they are needed.

 

I said full scale war – like Germany 1945. Even I still saw some ruins replaced by new buildings when I was a child. This was not simply done by bringing in goods from other countries.

After a global EMP it takes some months till things are running again but they will. It is just work.

Apocalyptic scenarios with tons of normal people changing to bad guys like shown in Walking Dead, Last Ship and other Shows are utterly BS.

 

Back in 1945 we weren't relying on EMP sensitive technology.

Constructing a building is considerably different from manufacturing modern equipment. Brick manufacture without the aid of technology has always been feasible but accessing any data that may have survived is impossible when the equipment necessary to do so requires other EMP sensitive equipment to manufacture.

Best case scenario computer technology would be temporarily restricted to a time when the components were completely manufactured by hand until we could rebuild the necessary equipment to make modern computers.

Worst case scenario the knowledge necessary to make modern computers would be on media that is inaccessible without said technology meaning you would have to reinvent the wheel of computer technology to access it again.

Agreed, but speed of recovery is considerably different depending on the scenario.

 

Right, building computers can be done much faster and is less effort than rebuilding 50% of a countries buildings which usually takes 1-2 years per house.

Anyway, it all can be repaired and replaced. This takes time but it will happen. It’s not the end of the world. After that you will be happy to have your backup (or you are one of those who starved to death)

 

Not to mention that the reconstruction of Germany took literal years. The same issues affected England, which was an utter disaster until into the 1950s. And those were both with loads of aid from other countries, which we absolutely can't rely on in the case of an EMP wreaking havoc on the globe.

 

We're not saying it's the end of the world. We're saying any data that survived would largely be worthless. After all what value would a company have if the product they made is no longer able to be used let alone distributed? Let's say you eventually managed to make use of the data again. Would your company even still be there?

 

A lot of non-hardened things will fail which causes chaos, problems and food shortage. That’s all. A lot of things can be simply saved by disconnecting them from the grid. Even most nuclear power plants should be safe.

No, your DEVELOPER DATA won’t be useless later. And most of other IMPORTANT data won’t be worthless too. (Science, Medical Data, Geographical Data, Statistics, History, Art etc.)

Pure business data of which a lot will be worthless within 10 years anyway will be of course worthless because this is kind of rubbish-data anyway.

 

It's entirely dependent on the data. Data that relies on computer hardware of a certain type would no longer have value if the hardware is no longer in existence. After all if you were Sony would you completely recreate your modern consoles or would you just start from scratch with a new design based on available resources?

Likewise data that depends on the presence of other data, like analytics data, save states stored on the cloud, and so on, would be completely worthless unless the company decided to make the game available again rather than create new games.

 

A massive solar storm would not destroy all PlayStations on earth.

 

You don't need to destroy every device. You just need to destroy enough of them that it's no longer financially worthwhile to resume manufacturing the device. If you were forced to reengineer much of the design (say, because the factories can no longer provide to the exact specification they previously were able to and you required) and manufacture new ones, you could just make new consoles. Which may lead to the previous games being incompatible.

 

Some unlucky PlayStations would be toasted. That’s all.

My mistake at the beginning was that a the EMP of a heavy solar storm could be compared to one cause by a nuke. But this is not the case. A solar storm can not kill electronics unless they are connected to the grid or another long line.

Google for E1, E2, E3 components of an EMP

 

Basically buy an uninterruptible power supply?

 

I don’t think that is enough.

 

Anyway, I use Dropbox and Visual Studio Team Services. But I have a cool leather jacket in case of the apocalypse.

 

And are you wearing if temperatures are rising during the apocalypse?

 

"The EMPs came a year ago since then millions have died and we are slowly rebuilding... Luckily I was prepared for such a thing and have backed up my entire unity project onto dvds. Time to continue my game. "

Honestly though does Unity backup to DVDs cuz if an EMP knocks out their data storage devices there's no point in having a unity project. Lol

 

As written above the real threat for electronics is an EMP by a nuke (not a solar storm) and this won’t be a global problem anyway. But still a threat for your data lets say for example by a terrorist attack.

 

Only if they are targeting multiple locations since any project of importance should have multiple locations it is stored in (data center for version control systems, backups made by said data center, local copies checked out of said version control system, and so on).

You're more likely to lose the programmer than the data he was working on in this scenario.

 

Anyway in any case, you can't imagine the nostalgia blackmarket that would raise if you are the lucky person who protected data and computer with faraday and other. And a computer is still useful after apocalypse, especially if you can't build another, making electricity is rather trivial even if not at the scale of a national power grid.

 

By the way I hope you weren't using the Home tier of CrashPlan...

https://www.theverge.com/2017/8/22/16184430/crashplan-home-shutting-down

 

I was, and the conversation got so derailed that you totally missed my followup that I'm moving to Backblaze. Actually, you participated in it when we were talking about Arqbackup.

 

Funny thing was I heard mention of CrashPlan removing Home on a YouTube video, immediately thought of you, searched because I couldn't remember where you had mentioned it, and then was shocked that it was in this thread.

Nothing like derailing a thread to the point I don't even remember the original conversation that was taking place.