Categories

    Amazon S3/Cloud Storage: How to configure cron job and Sync data (images, fonts, css and js) to Amazon S3

    Before you start syning files to S3, you need to setup con job in your server: 

    Please Go to Admin dashboard --> System Admin --> Tasks to get the command line that you need to setup cronjob (see below screenshot). You can setup cron via cpanel or ssh. Contact your hosting if you don't know how to do that.  Normally, the command line will be "wget -qO- 'https://domain.com/cron/task/run?key=3FSE@' &> /dev/null . Send this command line to your hosting if you want to ask them help you.

     

     

    Sync missing images, fonts, css and js to Amazon S3


    If S3 is enable and you find out some resources like css, font...are missing like the below screenshot. Please follow the below instructions to fix.

     

     

    Go to admin dashboard --> System Admin --> Storage System --> Manage Storage Services. Click on "Synchronise webroot directory", and you willl see below popup.

    Check Image, font, Cascading style sheets, Javascript then check the folders under "Webroot" that you have not syned/transfered to S3 before.

    For example in the below picture you will tranfer files (images, fonts, css and js) in "img" and "js" folder to S3

     

     

    Click OK to start Sync progress. You need to wait for a few minutes or may a few hours incase you have a lot of files that need to upload from selected folders to S3. The cronjob that you setup above will do this job so that you can close the web browser after clicking on "OK" button to sync. 

    Tranfer data to S3: In the setting page you will see a button called "Transfer", please click on this button if you enable S3 after site have launched long time ago to manually transfer all data such image, video... that uploaded by members before s3 is anabled to amazon.

    Clear caches: If you have any problems after s3 is enable. Clear caches and then try again. 

    Hope it helps.

    Share
    0% users marked this FAQ as helpful. Is this FAQ helpful?

    Yes No

Comments (14)

    View More
  • Gary and Rhonda
    Gary and Rhonda   What goes in this line that I've marked with green arrow? Is this where I put the bucket name? Example: mybucket.s3.amazonaws.com Can you show me an example of what to put here? Thanks!
    on 04/11/19 at 08:16 replyReply
    0 0 0 0 0 0 0
  • ketkew
    ketkew   Correct. Without CNAME, for example, images are served like this:
    https://s3.eu-west-3.amazonaws.com/yourbucketname/photos/thumbnail/2019/04/10/133/imagename.jpg

    If you want to have CNAME activated, it could be displayed like this (wh:
    https://cdn.yourdomain.com/photos/thumbnail/2019/04/10/133/imagename.jpg

    But besides S3 you need also activating AWS cloudfront (which also cost money) and upload your SSL certificate there and point the CNAME (cnd.yourdomain.com) to the cloudfront domain. At la...  more
    on 04/11/19 at 08:20 replyReply Edited
    1 0 0 0 0 0 1
  • Gary and Rhonda
    Gary and Rhonda   I already have cloudfront activated for S3 but what exactly goes in that box I marked with Green arrow? Please help and give an example screenshot or the example text of what goes there. Thank you I appreciate your help.
    on 04/11/19 at 09:16 replyReply
    0 0 0 0 0 0 0
  • ketkew
    ketkew   you should enter your CNAME there (ex. cdn.yourdomain.com)
    on 04/11/19 at 13:29 replyReply
    1 0 0 0 0 0 1
  • Gary and Rhonda
    Gary and Rhonda   In the script that you shared where it says ""Resource": "arn:aws:s3:::cloud.mydomain.com/*", --- What do I replace cloud.mydomain.com/* with? Does the "cloud" text stay or is it replaced? If replaced the with what? Thank you for helping... it just seems this should be a lot easier and clearer for setup of CDN and S3 for mooSocial.
    on 04/12/19 at 00:16 replyReply Edited
    0 0 0 0 0 0 0
  • ketkew
    ketkew   cloud.mydomain.com was the example aws bucket name. So the bucket name should be put there.

    and then the ip address part:
    if your server ipaddress = 1.1.1.1 then put 1.1.1.0/32 in the script.
    on 04/12/19 at 01:29 replyReply
    0 0 0 0 0 0 0
  • Gary and Rhonda
    Gary and Rhonda   So you mean to replace the last number of my server IP address with "0" and then follow with "/32" ?
    on 04/12/19 at 10:18 replyReply
    0 0 0 0 0 0 0
  • Gary and Rhonda
    Gary and Rhonda   After saving the bucket policy it shows this "This bucket has public access
    You have provided public access to this bucket. We highly recommend that you never grant any kind of public access to your S3 bucket." -- is this correct? Why have a policy that is not recommended by Amazon?
    on 04/12/19 at 10:36 replyReply
    0 0 0 0 0 0 0
  • Gary and Rhonda
    Gary and Rhonda   And after copying/pasting the COR script Amazon throws this error: "The XML you provided was not well-formed or did not validate against our published schema"---- this is the code I posted <?xml version="1.0" encoding="UTF-8"?><CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">;;<CORSRule><AllowedOrigin>*</AllowedOrigin><AllowedMethod>GET</AllowedMethod><MaxAgeSeconds>3000</MaxAgeSeconds><AllowedHeader>*</AllowedHeader></CORSRule>...  more
    on 04/12/19 at 10:39 replyReply Edited
    0 0 0 0 0 0 0
  • Gary and Rhonda
    Gary and Rhonda   Pasting it like this worked:

    <CORSConfiguration>
    <CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
    <AllowedHeader>*</AllowedHeader>
    </CORSRule>
    </CORSConfiguration>
    on 04/12/19 at 10:45 replyReply
    0 0 0 0 0 0 0
  • Gary and Rhonda
    Gary and Rhonda   I'm still at this point of not getting any data to sync to S3 bucket -- I appreciate your help ketkew -- so many variables here to figure out what the problem is.

    I've also setup the sync cron command and my webhost says the command line is working without errors, but it's not pushing them to S3. In the task panel it shows Aws Task Transfer succeeded:never... I don't know if that's because my S3 bucket is not setup right or if the cron is not working (although my webhost provider says it is wo...  more
    on 04/12/19 at 11:36 replyReply Edited
    0 0 0 0 0 0 0
  • Gary and Rhonda
    Gary and Rhonda   It's also not clear per the mooSocial instructions of what folders to sync. Do we select all of them? If I went by what the image shows per instruction with this topic it says to "only" check the images and js folders.???? I apologize for the many questions but this should be sooo clear.. mooSocial help us please.
    on 04/12/19 at 12:24 replyReply
    0 0 0 0 0 0 0
No login
format_indent_increase