Configure Flmngr file manager to use Amazon S3 storage

Amazon Web Services (AWS) are very popular. Many websites use Amazon S3 for storing files and images there for fast serving media available to visitors.

Flmngr file manager can use Amazon S3 bucket with files and here are instructions on how to configure it.

Install Flmngr backend as usual and pass the steps below after that.

Basics of using network filesystems with Flmngr

Flmngr uses its backend installed on your server to serve files. Usually, it serves local disk filesystem files, but this is not a rule.

Being filesystem-agnostic Flmngr can use any filesystem you give to it. So Flmngr can use any filesystem you want: from NFS or Samba to Amazon S3 or Azure Blob.

You just mount the remote filesystem as a regular directory and specify it as a local disk filesystem directory. So Flmngr will use this filesystem for content managers (who edit website contents) in editing mode, but all the URLs will be generated using direct links to S3.

The process of configuring S3 with Flmngr file manager is simple:

  • Mount S3 bucket into the local filesystem
  • Specify a mounted directory to Flmngr backend (to access files by content administrators)
  • Specify URL prefix for the client (to generate public URLs)

This article will show you how to configure this in PHP & Unix environment, but you can use another backend, the difference will be trivial.

How to mount S3 bucket

Install s3fs command line application

Amazon S3 bucket is like some disk with the filesystem. It contains a directory tree and files inside it.

AWS provides API for accessing it, so you can do whatever with those files from your server where you install the file manager backend.

Mounting S3 bucket under Unix operating systems (Linux, BSD, MacOS, etc.) can be done using s3fs app which is a part of FUSE (filesystem in userspace).

Install FUSE by running the console command, for example in Debian/Ubuntu this is:

sudo apt install s3fs

Full list of install commands for different OSs is on GitHub of s3fs.

Create a directory for your storage

After you installed s3fs, create somewhere a directory to where you wish to mount a bucket. This will be a mount point in the terms of Unix file management subsystem.

Create a file with access credentials

Go into Amazon S3 web console and create there some user with a policy to access and modify all the objects (files) in the bucket.

Copy a key ID and secret access key for this user and put them into some file fuse-s3.passwd in the format (single line):

ACCESS_KEY_ID:SECRET_ACCESS_KEY

You need to restrict unauthorized access to this file, so please make sure it has 600 permissions:

chmod 600 fuse-s3.passwd

Add S3 bucket mount point using fstab

If you want to create a permanent mount point in your OS, you need to modify your /etc/fstab file. Add the line in this file:

s3fs#flmngr-test-bucket /var/www/files fuse _netdev,uid=www-data,gid=www-data,nonempty,umask=077,use_cache=/root/cache,passwd_file=/etc/fuse-s3.passwd 0 0

or the variant with more wide access permissions:

s3fs#flmngr-test-bucket /var/www/files fuse _netdev,uid=www-data,gid=www-data,allow_other,nonempty,umask=000,use_cache=/root/cache,passwd_file=/etc/fuse-s3.passwd 0 0

Let's go through all the options:

  • s3fs#flmngr-test-bucket - the name of your mount point. Use it to administer it in the future by passing to mount and umount utilities if you need. flmngr-test-bucket is here your bucket name, copy it from Amazon S3 web console.

  • /var/www/files - the directory where to mount the bucket to. Be sure this directory exists, is empty.

  • uid=www-data,gid=www-data - the user and group of your server who will be the owner of the mounted bucket. Use here a user and group you run your webserver (Nginx/Apache/...) or webapp (Node/...) under.

  • allow_other - a flag for giving access to this directory to other users. Remove it if you do not need such option.

  • nonempty - a flag to mount the bucket into a directory even this directory is not empty. Very useful especially if you use GIT in your workflow to push files to the server and unable to commit empty directory there without placing some .keep empty file inside (GIT does not track empty directories).

  • umask=000 - umask for accessing the files (permission bits to reset, subtracted from actual permissions). 000 means full access for owner, group and others, 077 means full access for the owner only.

  • passwd_file=/etc/fuse-s3.passwd - location of the file with access credentials you created in the previous step. Place fuse-s3.passwd file inside /etc/ to let s3fs find it. Do not forget to set correct permissions (600) for it.

Now your bucket will be mounted every time you boot your server. You can mount it without rebooting by typing in console:

sudo mount s3fs#flmngr-test-bucket

and the command for unmounting:

sudo umount s3fs#flmngr-test-bucket

Does not work? See troubleshooting chapter.

Configure Flmngr

The main idea is to synchronously set paths both on the client and on the backend:

Client side configuration: prefix for URLs
Files URL (Flmngr.urlFiles parameter)
https://flmngr-test-bucket.s3.us-east-2.amazonaws.com/
/var/www/files/
Server side configuration: location of mounted S3 bucket
config['dirFiles'] parameter in backend config
+ /path/to/image.png

Read the details of this scheme below.

Configure Flmngr backend

In order to explain to Flmngr server part where files are located, find a place where you call Flmngr FlmngrServer::flmngrRequest request. The config inside has to be changed.

Specify dirFiles from there to lead to the directory you mount your S3 bucket into. /var/www/files in this case.

dirTmp and dirCache are configured as usually: specify some private directories on your server for them (public web access is not required).

Please use absolute paths wherever this is possible to avoid common misconfiguration problems.

Configure Flmngr client

Configure Flmngr client

Flmngr backend works with S3 filesystem only when your content manager guy edits some article or media. But in order to let Flmngr generate correct public URLs for website visitors, please specify "Files URL" property in Dashboard or Flmngr.urlFiles parameter if you pass config manually using API.

The value should be like this:

https://flmngr-test-bucket.s3.us-east-2.amazonaws.com/

These values required to generate this URL can be found in your AWS console:

flmngr-test-bucket - a name of your bucket us-east-2 - a region of your bucket

You can also subtract this path by checking some file from S3 web console and pressing "Copy URL" button above the files.

This value will be used as a prefix for the path returned by Flmngr backend.

For example, if you mounted a bucket into /var/www/files and uploaded a file into /var/www/files/cars/auto.png, file manager backend will return cars/auto.png as file path. Flmngr client will create the URL https://flmngr-test-bucket.s3.us-east-2.amazonaws.com/cars/auto.png for using on public pages. This URL is handled by Amazon S3 directly (Flmgnr backend bypassed here).

Configure access policy

Troubleshooting and lifehacks

Unable to mount S3 bucket or problems accessing a mounted bucket

Try mounting it manually using this command:

s3fs flmngr-test-bucket /var/www/files -o passwd_file=/etc/fuse-s3.passwd -o dbglevel=info -f -o curldbg -o nonempty -o mp_umask=0000 -o umask=0000 -o allow_other

Please see the description of the parameters above, they must be the same you wrote in /etc/fstab to debug exactly the same configuration.

You will see logs in console output while mounting and accessing the bucket. Ctrl+C will stop this app and unmount the mounted bucket.

Note: running this command with -o allow_other from non-root user is possible only when you uncomment the line user_allow_other in /etc/fuse.conf.

Do not want to have a permanent mountpoint in /etc/fstab or do not have access to it

You can create a mount script for your S3 bucket by the command from the question above, just remove -o dbglevel=info -f -o curldbg part from it to run the process in background.

You can unmount this bucket later by passing the target directory to umount:

umount /var/www/files

I can use S3 bucket in Flmngr, but files are not accessible from the web by website visitors

First of all, try to paste some generated image URL directly in the address bar of your browser. What do you see?

If you see some syntax problems or any errors except 403 error, you should check your "Files URL" or Flmngr.urlFiles property passed to Flmngr file manager client.

If you get HTTP error 403 ("Access denied") from Amazon, please check you have opened public access to this bucket:

  • Go to the bucket in Amazon web console, select your bucket, go into the "Permissions tab"
  • Uncheck the "Block all public access" parameter
  • Add bucket policy like:
{
    "Version": "2012-10-17",
    "Id": "Policy1637840001466",
    "Statement": [
        {
            "Sid": "Stmt1637839995898",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::flmngr-test-bucket/*"
        }
    ]
}

You can use this generator to generate a policy for your bucket.

Flmngr file manager shows empty storage

Probably you created a directory for mounting a bucket but did not mount it. Or your bucket is really empty (check do you use the correct bucket if have a number of them).

I can see images by direct URLs in the browser, but Flmngr can not access them

Flmngr can return a message about "Root directory is not accessible", which means it can not access S3 bucket using API.

  1. Check your mount point directory is accessible by your application. Run something like:

    sudo -u www-data ls /var/www/files

    where www-data is the user you run the app under and /var/www/files.

    If this command fails, the issue is related to your local server permissions. Especially check mount options here.

  2. If server permissions are fine, but your Flmngr backend can not access S3 bucket, please check you have applied a policy that allows to list/read/write into this bucket to the user you specified as the user for the mount point. If you see files but can not change them using OS tools, the problem is just in write access, if you do not see any files or mounting fails — please check the policy is attached to the user.

I have different users which have subfolders in one bucket

The configuration for multiuser applications with S3 is the same as for local storage: just add some subfolder both for config['dirFiles'] on the server and to Flmngr.urlFiles in client config (yes, you don't need to use Dashboard parameter because it needs to be dynamic).

For example on the server:

FlmngrServer::flmngrRequest(
    array(
       'dirFiles' => '/var/www/files/' . $user.dir . '/', // $user is your variable with 
                                                          // some info about authenticated user
       'dirTmp'   => '/path/to/tmp',
       'dirCache'   => '/path/to/tmp',
    )
);

On the client:

    var dirUser = "foo"; // generate it on the server side by $user.dir when you output page HTML
                         // or read from Cookies or some variable related to user (it depends on your CMS)
    CKEditor.replace("editor", {
        Flmngr: {
            urlFiles: "https://flmngr-test-bucket.s3.us-east-2.amazonaws.com/" + dirUser + "/"
        }
    });

All works fine except ImgPen image editor: it shows a blank image

Looks like you did not configure CORS for S3 bucket correctly. Probably you also have a related message in your browser console.

In Amazon web console go to the bucket and switch to the "Permissions" tab. There is a section "Cross-origin resource sharing (CORS)" there, please paste a JSON with origin (a protocol + domain + port you run your image editor from), "*" is for any origin.

For example:

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "HEAD",
            "GET"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": []
    }
]
CORS configuration