Google Cloud Storage FUSE (opens in new tab)

Your repository seems to have both an Apache and MIT license. What license are you distributing your code under?

Edit: I see you said it’s dual licensed. From the look of it both allow Google or any other company to copy and reuse code, so what are you upset about?

Hi Ofek,

I am a contributor who works on the Google Cloud Storage FUSE CSI Driver project. The project is partially inspired by your CSI implementation. Thank you so much for the contribution to the Kubernetes community. However, I would like to clarify a few things regarding your post.

The Cloud Storage FUSE CSI Driver project does not have “in large part copied code” from your implementation. The initial commit you referred to in the post was based on a fork of another open source project: https://github.com/kubernetes-sigs/gcp-filestore-csi-driver. If you compare the Google Cloud Storage FUSE CSI Driver repo with the Google Cloud Filestore CSI Driver repo, you will notice the obvious similarities, in terms of the code structure, the Dockerfile, the usage of Kustomize, and the way the CSI is implemented. Moreover, the design of the Google Cloud Storage FUSE CSI Driver included a proxy server, and then evolved to a sidecar container mode, which are all significantly different from your implementation.

As for the Dockerfile annotations you pointed out in the initial commit, I did follow the pattern in your repo because I thought it was the standard way to declare the copyright. However, it didn't take me too long to realize that the Dockerfile annotations are not required, so I removed them.

Thank you again for your contribution to the open source community. I have included your project link on the readme page. I take the copyright very seriously, so please feel free to directly create issues or PRs on the Cloud Storage FUSE CSI Driver GitHub project page if I missed any other copyright information.

You licensed the code as MIT - https://github.com/ofek/csi-gcs/blob/master/LICENSE-MIT

Are you saying you have an issue with them copying your MIT licensed code?

Are you accusing Google of "large part copied code" based on an old commit which is not even used in this official launch? Do you have any evidence from their recent commit? At least I don't see the current two repos are anywhere alike, except for that you both implement the same interface. Also, they did reach out to you and you just didn't respond, so why are you complaining now?

It makes me sad that no one here cares about whether your blame is true. And I'd expect you can provide more convincing evidence. But looks like the accusation is not even true. It's not fair for those contributors man, I hope you can apologize.

Update: attribution has been added to the readme file https://github.com/GoogleCloudPlatform/gcs-fuse-csi-driver/c...

Haha a lot of funny comments here. I think overall it's neither here nor there. You should be proud that the "elites" at Google copied your code ;)

This seems overly pessimistic to me.

Sure you're not going to use this as a consumer in place of a local disk, nor are you going to use this as part of your web app.

But there are lots of situations in reporting, batch/cron jobs, data processing, and general file administration where it's incredibly easier to use the file system interface than to use an HTTP API via a cloud storage library. Which FUSE is a godsend for. The latency doesn't matter in these cases for one-off things or scripts that already take seconds/minutes/hours anyways.

So no this isn't niche or a toy. It's a fantastic production tool for a lot of different common uses. It's not for everything but nothing is. Use the right tool for the job.

Yep. I once inherited a system where the previous team had used GCSFuse to back the `/etc/letsencrypt` directory on a cluster of nginx webservers. It "worked" and may have been a reasonable approach at the time they built it, avoiding setting up a single "master" to handle HTTP-01 challenges (and it was before GCP's HTTPS LB could handle more than a handful of domains/certificates). The problem was that as the number of domains/certificates it handled increased, nginx startup or config reload time got slower and slower as it insists on stat-ing and reading every single file in that directory in the process. It got high enough that it started running into request throttling on the storage bucket. It's no fun when `nginx -s reload` takes two minutes and sometimes fails completely.

>What I would really like to see is a two-tier cache system

Is there any sort of Linux HSM (Hieracrhical Storage Manager)? I haven't see any and have been a bit surprised nothing has really developed there. They can manage putting hot data in RAM, SSDs, colder or larger data on spinning rust, deep freezing onto a tape silo or a cloud storage...

Some of the NAS devices and RAID cards can support a two-tier caching or data migration using SSDs, where hot or highly-random data (usually identified by smaller write sizes) go to the SSDs, and then can migrate to the spinning discs.

I've done some "poor mans" version of this using LVM, where I can "pvmove" blocks of a logical volume between spinning discs and SSDs, which is pretty slick, but a very crude tool.

I once evaluated using s3fuse for managing about 36 million images. The old storage model was on a filesystem so it was supposed to make a smooth transition to the cloud.

AWS Premium Support wisely advised me against it, not just because of latency but also because the abstraction makes /far/ more API calls then a native solution would.

After a bit of testing to confirm, I switched to using native API calls. That code was easy to write and the performance was great. I've been wary of cloud FUSE adapters ever since.

I'm working on optimizing FUSE using eBPF (ExtFUSE [1]) and adding a caching layer exactly as you mentioned. Will post publicly when ready.

1. https://github.com/extfuse/extfuse

> What I would really like to see is a two-tier cache system: most recently accessed files are cached to RAM, with less recently accessed files spilling over to a disk-backed cache. That would open up a world of additional applications whose useful cache size exceeds practical RAM amounts

This is really hard to get right if the origin cloud storage is anything other than immutable. Otherwise you're in for a world of cache invalidation and consistency pain.

I've gradually come round to the other opinion: there should be devices that sit on the PCIe/NVMe bus and provide a blob storage API rather than a block one, and there should be an operating system blob API that is similar to but not identical to the filesystem one.

Same experience. I remember opening a .docx in Word and watching it hang or studder at different operations. I think you'd need very reliable and low latency networking for this to be anything but a painful to use toy.

I'd be curious to see how it works running on EC2, especially with an S3 endpoint in the VPC. Although I still think you'd be better suited by using S3 as an object store, given the option to built it right.

Catfs is not super production (there are some small changes you need to make in inode handling), but you can do this. We have it on top of goofys. They both need a few changes to work under load but what we do is quite standard:

1. Goofys for S3 FUSE

2. Catfs for local disk caching

3. Linux caches in memory

4. Mmap file means processes share it

5. One device then exports this over the network to other machines, each of which have an application layer disk cache.

6. Machines are linked via 10 GigE (we use SFP+).

Overall the goofys and catfs guy (kahing) wrote very performant software. Big fan.

> most recently accessed files are cached to RAM, with less recently accessed files spilling over to a disk-backed cache

Isn't this how most servers run normally? (parts of) files which are accessed are in page cache, the rest is on "disk"

That page shows a `mkdir` is 3 json commands. I wonder if its that many HTTP requests.

>>> every file system operation is fundamentally an HTTP request, so the latency is several orders of magnitude higher than the equivalent disk operation

gcsfuse latency is ok as it embodies "infinite sync & persistence" ;)

Well, and there's no such thing as opening a file and modifying some small part of it. That's emulated with a full rewrite of the whole object.

Uh, how does it perform from a Google Compute Engine Virtual Machine?

If it performs well there, I could imagine that being pretty useful.

Moreover, there is no SLA on those FUSE adapters so putting it into any part of production is too risky.

My personal conspiracy theory: most "cloud services" are just... bad.

VMs and disk space I understand completely, having machines on-prem is too much of an hassle and the price isn't that bad. But for stuff like this, managed services, databases especially, you're just getting scammed.

Looking at change descriptions, looks like underlying changes were made to get to this like now using GO client library. I would expect a more stable product, and better performance which looks like the performance benchmarks located under docs has been updated as well. Happy to finally see Google standing behind this, and the official CSI driver is really cool to see.

Heh, my old laptop has a git clone of this from September 1st 2016.

yes

> export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`

How do Googlers access Google Drive from their Linux workstations? Do they have an internal GDrive client?

Unfortunately it's common to have a policy in place disallowing 3rd-party app api access to drive storage. This prevents apps like rclone from working, but the drive client works because it isn't 3rd-party.

GCS is not Drive, it's Google's equivalent of S3.

https://github.com/astrada/google-drive-ocamlfuse is one option

Use rclone.

JuiceFS is mostly POSIX compatible, but there are important caveats such as no extended ACL, copying files changes their mtime (impacts backup tools), it offers "close-to-open" consistency (dangerous for log appenders), etc.

Choosing an appropriate solution in this space still depends on what you need to do with the storage, and a few other options are MooseFS (https://github.com/moosefs/moosefs) SeaweedFS (https://github.com/seaweedfs/seaweedfs) Curve (https://github.com/opencurve/curve) GeeseFS (https://github.com/yandex-cloud/geesefs)

One challenge with writes in the middle is that it changes the file hash. Cloud services typically expose the object hash, so changing any bit of a 1TB file would require a costly read of the whole object to compute the new hash.

You could spilt the file into smaller chunks and reassemble at the application layer. That way you limit the cost of changing any byte to the chunk size.

That could also support inserting or removing a byte. You'd have a new chunk of DEFUALT_CHUNK_SIZE+1 (or -1). Split and merge chunks when they get too large or small.

Of course at some point if you are using a file metaphor you want a real file system.

pretty standard limitation of object storage services iirc

This works, there is nothing stopping it, but just like all cloud object storage it will trigger a complete re-write of the object when saved.

Using FUSE doesn't cost you anything extra, but it doesn't make the underlying storage free.