I usually have a couple cron jobs scheduled, executing backup scripts between EC2 instances and store the data on S3 via Python's Boto library. Everything was running smoothly for several months until one day, the script was failing to authenticate with a 403 error from AWS S3, on a specific instance.
Ports are open, credentials are okay and nothing has changed. What could possibly go wrong?
Many hours of searching for possible sources for the problem ended. The last thing on my mind was a possible issue with time settings. After looking into that a bit more, it turned out that a recent update on Boto's authenticate which made EC2 connections use hmac-v4 by default. Update referenced in this commit.
The hmac-v4 method is more sensitive to clock skew. If an instance's clock differs from the internet standard time servers by more than a couple minutes the authorisation will be rejected.
All you really have to do is sync up the time on your server All you have to do is setup the ntp daemon on the server. It sets and maintains the system time of day in synchronism with Internet standard time servers.
If you're on Ubuntu, you can just run these 2 commands and everything should be in place.
sudo apt-get install ntp
sudo ntp -u ntp:ntp
It's amazing how some simple issues, get so much frustrating and time consuming to solve, mostly due to overlooking some details that can be taken for granted. Hope you found this post helpful!
