Uploading large files

Introduction to File Browser and background

There is a web server written in Go called File Browser. For a while, I was using MinIO to upload files, but it takes a little extra work to use files uploaded to MinIO directly from another application.

File Browser can display a list of files, as its name suggests, but it also seems to be able to be used as a simple uploader. It is a popular project, with over 26K stars on GitHub, but API documentation is not currently available.

Uploading using Node.js and implementing it in Ruby

You can make some assumptions using the network tab in your browser, but fortunately, the method for Node.JS was left in an issue.

Using this code as a reference, I wrote a simple file uploader using HTTParty:

class Filebrowser
  include HTTParty

  def self.auth(username, password)
    headers = { "Content-Type" => "application/json" }
    body = { username:, password: }.to_json

    post("/api/login", headers:, body:).then do |res|
      new(res.body) if res.success?
    end
  end

  # Store the logged in token
  def initialize(token)
    @token = token
  end

  # If you want to save to `/Media`, specify `/api/resources/Media`
  def upload(file_path, mime_type = "text/plain")
    url = "/api/resources/Media/#{File.basename(file_path)}?override=true"
    headers = { "Content-Type" => mime_type, "X-Auth" => @token }

    self.class.post(url, headers:, body: File.read(file_path))
  end
end

The problem with large files

This uploader works fine for relatively small files. But what if the file you want to upload is over 1GB?

First, I prepared a file called video.mp4 with a size of 2.8 GB and uploaded it.

As you might expect, this program uses 2.8 GB of memory.

If you have plenty of memory like in a desktop environment, there won’t be much of a problem running this program, but if you run this program on a server with limited resources, it will be affected by the size of the file you upload.

A solution using stream processing in Ruby

Node.JS has fs.createReadStream(), but what should we do in Ruby?

class Filebrowser
  def upload(file_path)
    url = "/api/resources/Media/#{File.basename(file_path)}?override=true"
    headers = { "Content-Type" => "application/octet-stream",
                "Transfer-Encoding" => "chunked",
                "X-Auth" => @token }

    self.class.post(url, headers:, body_stream: File.open(file_path, "rb"))
  end
end

I specified Content-Type: application-octet-stream and Transfer-Encoding: chunked in the headers, and specified body_stream instead of body.

I uploaded the exact same file, but I was able to successfully reduce the overall memory usage to 23.3 MB.

I think this script can be run without any problems regardless of the size of the file.