OneDrive alternative that can handle more than 400.000 files
Hi, I have been using OneDrive for a long time and like it a lot, but over the years, especially in recent months, I have noticed that some files are not synced properly and OneDrive is using a lot of resources (CPU and RAM). After running a diagnostic, I…
Hi, I have been using OneDrive for a long time and like it a lot, but over the years, especially in recent months, I have noticed that some files are not synced properly and OneDrive is using a lot of resources (CPU and RAM). After running a diagnostic, I found that many files are not synced at all and OneDrive is stuck on “Looking for changes” and never finishes. I read that Microsoft recommends keeping the number of entities under 400,000, otherwise OneDrive may not work reliably. My stats are around 350,000 files and 400 GB, but I use about 80 GB, other is on-demand. As I need all these files, I am thinking of migrating to another solution. I have a server at home and enough space. What software would you recommend? I tried OpenCloud, but the Windows UI is outdated and often simply decides it will not sync certain files. What app can handle more than 400.000 files? I mostly need just to sync files, to have a backup somewhere else. I don't need fancy UI or a lot of sharing features.
Collected discussion
Expand the replies to this comment to learn how AI was used in this post/project.
I work a lot without internet. And then I access files via web UI, and have another computer that has the same setup regarding files for easier access.
Yeah, that is a deal breaker...
Hm, I will try it. But to be honest, yes it is better that the files are projected to cloud storage, so you could access them without cloud provider. If for example the software becomes broken.
That sounds nice, but as I looked it is peer-to-peer, so I would run it on the server and on my clients... That could work ...
How is the client? Does it bug out and fail to sync files? And does the thumbnails service work properly? Sometimes the desktop app just freezes.
Absolutely everything, Images, movies, school work, work files, personal projects, documents, books etc. I think the problem is my approach rather than an app. .
Yes, that is the solution. I just need to change my workflow and reorganize everything.
Thank you so much for your story! And advice
The real reason you're hitting that 400k limit is exactly what you mentioned: syncing node_modules, .git, and Flutter build folders. No sync engine handles hundreds of thousands of tiny, constantly changing files well. Let Git manage your code (you can easily self-host GitLab or Gitea for this!). Use your self-hosted cloud (like Nextcloud or Seafile on your home server/VPS) strictly for actual documents and media. Your entity count will drop drastically, and sync will run flawlessly.
I moved to opencloud and Im quite happy with it. does what I need easy to manage and upgrade than nextcloud. I don't know on the 400k claim as I'm not there yet but I sync around 500gb of raw photos without much issues
Why not standup a NAS and then access it with whatever file share tool works best for a given client. Seems like the obvious selfhosted solution.
I mean if you just want 2 repositories to sync up then syncthing would work perfectly. But if you want a full cloud replacement then I recommend Nextcloud. has much newer look and interface then owncloud and a lot of add-on features. Pro/Con It updates allot. Difficult first setup and tuneing. Rock solid once it is setup.
Oh hey, I was in your position a few years ago :) I bet you do a lot of js programming (node_modules bs) that's why its 400k files haha. I would just use backup approach not sync, you'll have a lot of issues with the syncing approach with such amount of files for probably all software. And a word for those who suggest SyncThing, are you really sure you have such amount of files like OP because I bet you do not, because if you did, you would not suggest such a very slow and buggy and clunky application. OP, DO NOT USE SYNCTHING, IT IS HORRIBLE. Since I mentioned I was in your position a few years ago, I use the backup approach, in my case I don't selfhost it, but you can def selfhost with it. I settled with Kopia UI and I backup regularly. You can use Kopia Server on your infra and then Kopia UI on your main workstation. It's really fast at least in my case, and I have more files than you. You can also "ignore" the folders you don't want (like node_modules). You can run schedule with Kopia UI to let it backup automatically. There's also NextCloud as others suggested it. but a word of advice, do not update it regularly, it breaks :)
Syncthing. But as far as I know you cannot sync on demand, you would need full 400 gb space locally.
Local file servers?
OneDrive is the reason I started learning how to manually back up my info. It works most of the time and the office interactions are pretty convenient, but when it breaks holy shit it breaks. I mainly use Rclone now, It is a feature-rich alternative to cloud vendors' web storage interfaces and supports more than 70 different cloud providers, including Onedrive. It is very simple to use once you get the hang of it, predictable, loggable and easy to automate with cron jobs. If you have enough space at home, it's easy to turn a home server into a storage backup with rclone provided you´re not behind CGnat
I've definitely synced more files with SyncThing, but it was a pure background process so I didn't pay any attention to it other than "it worked".
Maybe try seafile. Pretty easy to set up with docker and I guess you can easily test if the sync clients can handle it or not. However be aware that the seafile backend stores its data in some kind of block storage and not directly mirrored to disk. I don't think this is an issue, but some selfhosters don't like it. Also be aware that it keeps a history and trash, so if files in your setup change super often maybe disable it.
If you have a server at home, do a samba drive on it and connect to that from your PC. That's more as a "on demand" approach for everything. If you strictly want a backup solution, I'd recommend Kopia. I personally do both, I use samba to access the files on my home server and Kopia to back them up to a 10TB external server. At the moment it's handling ~450k files at ~400gb total (images, videos, pdfs, etc.) Before Kopia I tried syncthing, duplicati, nextcloud. None were exactly in-line with what I needed or I found them to be very hard to set up (compared to Kopia for example). Syncthing randomly decided to not work and required restarts (still have no idea why) You can set Kopia to sync files anywhere from every minute to every year (and everything in between).
OpenCloud is your best bet. Since it’s based on OCIS it is used by CERN and can handle this amount of data easily. They run petabyte deployments over there.
my nextcloud instance is slightly under your total count, using about 400GB and rarely ever has issues syncing. i think the last one was a ridiculously long mp3 file path to a linux host. surprisingly it wasn't a king gizzard track (i think it was non-ascii characters hurting the length but it's been so long) i rarely need to use the UI and haven't used opencloud but if that's a problem i doubt NC will be much more fun.
Nextcloud
What are all these files and how are you using them? Your use case is more helpful to solving the issue holistically. Let’s solve for the real problem X versus onedrive problem Y.
I replied with Seafile but also with checking out Copyparty. I will warn that the UI is pretty hideous. But it works remarkably well.