AWS, Cloud, DevOps & Cloud Solutions

How to Mount S3 Bucket on Linux (AWS EC2 Instance)

USe Case:

AWS S3 is awesome resource for cloud object storage and the consumption of S3 is varies from customer, very common use cases were:
Backup and Storage – Provide data backup and storage services for others.
Application Hosting – Provide services that deploy, install, and manage web applications.
Media Hosting – Build a redundant, scalable, and highly available infrastructure that hosts video, photo, or music uploads and downloads.
Software Delivery – Host your software applications that customers can download.

Lets  have an application  logs in AWS EC2 application cluster which have 4 instance as part of the cluster, from each application server logs need to stored centrally and need to accessible (Read-Only) frequently from all server.

We can consider NFS sort of solution, even now we have EFS from Amazon but it costly and even the same data were using for their analytics solution. So we thought to use S3 to satisfy both the requirement.

We have mount S3 on all required Linux EC2 instance using S3fs, so that all required instances have access to logs at the same time their analytic solution also can read data using s3api.

Filesystem in Userspace (FUSE) is a simple interface for userspace programs to export a virtual file-system to the Linux kernel. It also aims to provide a secure method for non privileged users to create and mount their own file-system implementations.
S3fs-fuse project is written in python backed by Amazons Simple Storage Service. Amazon offers an open API to build applications on top of this service, which several companies have done, using a variety of interfaces (web, rsync, fuse, etc).

Below are the pre-requisites to install and setup S3fs:

EC2 instance with root access or sudo access to install S3fs and mount volume
IAM user which have S3 Full access (For Upload/Download)
Download latest S3fs package from http://code.google.com/p/s3fs/downloads/list
update your system to latest using
yum update all (for CentOS)
apt-get update (for Ubuntu)
Install below dependencies before installing S3fs package
For CentOS
yum install gcc libstdc++-devel gcc-c++ fuse fuse-devel curl-devel libxml2-devel openssl-devel mailcap
For Ubuntu
apt-get install build-essential gcc libfuse-dev libcurl4-openssl-dev libxml2-dev mime-support pkg-config libxml++2.6-dev libssl-dev
Follow the below steps to mount your S3 bucket to your Linux Instance:
Step 1: Download latest s3fs package and extract:

wget https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/s3fs/s3fs-1.74.tar.gz
tar -xvzf s3fs-1.74.tar.gz

Step 2: Update OS and install dependencies as mentioned in above pre-req.

Step 3: Now change to extracted directory, compile and install s3fs source code.

cd s3fs-1.74

./configure –prefix=/usr

make

make install

Step 4: Use below command to check where s3fs command is placed in O.S. It will also confirm whether installation is ok:

which s3fs

Step 5: Get IAM user Access and secret key which have appropriate permissions (e.g. S3 Full access), You can get the same from AWS IAM console

Step 6: Create a new file in /etc with the name passwd-s3fs and Paste the access key and secret key in the below format and change the permission for the file:
echo “AccessKey:SecretKey” > /etc/passwd-s3fs
chmod 640 /etc/passwd-s3fs

Note: Replace AcessKey and SecretKey with original keys.

Step 7: Now create a directory and mount S3bucket in it. Here, Provide your S3 bucket name in place of “your_bucketname”

mkdir /jagadishcloudarch

s3fs your_bucketname -o use_cache=/tmp -o allow_other -o multireq_max=5 /mys3bucket

Replace your_bucket = S3 bucket name which you want to mount
Replace /jagadishcloudarchS3 = Directory name which you want to mount
/usr/bin/s3fs jagadishcloudarch -o use_cache=/tmp -o allow_other -o multireq_max=5 /jagadishcloudarch
You can validate whether it is mounted using below command:

[root@ip-172-31-49-68 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 488M 56K 488M 1% /dev
tmpfs 497M 0 497M 0% /dev/shm
/dev/xvda1 7.8G 1.1G 6.6G 15% /
s3fs 256T 0 256T 0% /mys3bucket
s3fs 256T 0 256T 0% /jagadishcloudarch
[root@ip-172-31-49-68 ~]#

Note: At any given point you can unmount this volume using below command:
umount /jagadishcloudarch
Now this volume is Non-Persistent i.e; once you reboot your system this mount point wont exists, to make it persistent and automatically mount for every reboot we need to add below entries to /etc/rc.local

nano /etc/rc.local

Add below Line and save the file:
/usr/bin/s3fs jagadishcloudarch -o use_cache=/tmp -o allow_other -o multireq_max=5 /jagadishcloudarch
Now you should be able to read and write filed to S3 (Considering you have S3 full access).

touch /jagadishcloudarchS3/jagadish

[root@ip-172-31-49-68 ~]# ls -lrt /jagadishcloudarch/
total 1
-rw-r–r– 1 root root 0 Jun 7 10:11 jagadish
[root@ip-172-31-49-68 ~]#

About The Author

Leave a Reply

*