Exposing your AWS access keys on Github can be extremely costly. A personal experience.

Recently, I was doing a Ruby on Rails course on Udemy that involved opening an AWS S3 service and creating some buckets for develpement and production so images and files could be stored on AWS instead of locally on my computer.
I had already used S3 when I did Mattan Griffel’s OneMonthRails a few years ago. I deleted those buckets and credentials and created two new buckets and access keys.
After the csv file was dowloaded with access key and password, I double clicked it, and it was opened by Atom, the text editor I was using. Since I had already copy pasted the access codes in my .env file which I listed in gitignore, I went ahead and pushed to GitHub as was required by the course.
This, I later reaised, was a big mistake, as I should have opened it with text editor so it remained on my desktop.

The Shock
Sometime later, I got an email from Github’s GitGuardian informing me that the API keys are exposed and to confirm if it was true. I was about to ignore it and tell them they were wrong. But I decided to go to my repo and check. Sure enough, my csv file was there with the access key information. I tried all kinds of things to rollback the commit I had made, then found that I could delete the file with an ‘rm’ command, did that.
That wasn’t enough.
Soon I received an email from Amazon to inform me that the account has been compromised and I should take action. Panicking, I went to the billing section to see there was a charge of over $3000 to date, and the projected cost was about $15000! I nearly had a heart attack.

The people who put AWS in AWSOME

I emailed Amazon explaining my situation. Went again to AWS site, wrote another email and fortunately clicked the telephone icon in the submit field instead of the email option, which resulted in them calling me.
I called them back, and the friendly staff transferred me to the right people to help me. Spoke to Stephanie who allied my fears saying it will be all right, don’t panic, and put me onto a gentleman called Will who took me step by step through the whole process,explaining what happens when you leave your credentials exposed. The call lasted over 15 minutes, I must admire the fact he never lost his cool even once. Every question was answered, every doubt, however silly, was cleared.
What happens when you expose your access keys?
Will said that a lot of novice programmers tend to push code with sensitive information, and this is not the first time. Since Github is a public and open source site, malicious users leave a programme with a script to scan every commit and push that’s made to Git for exposed credentials. Once they find such keys, the malicious user immediately uses them to his own ends. Apparently a lot Bitcoin dealings and other transactions are done illegally using such exposed keys. While Will was walking me though the steps, I saw instances running in every region, from Sao Paulo to Singapore (instances are, I’m told, computer spaces rented by Amazon to users). And the rent is humongous if used on a large scale basis, running for hours. Will helped me shut down all the running instances. I asked if I should close the account. ‘No’, was the answer, as the keys will be still used, we had to shut down the instances one by own going through the regions. Earlier, I had closed all these running regions but that wasn’t enough as each had to be terminated.

How to guard your Amazon access keys like a state secret?
While I was lamenting the lack of morals and sympathy and wondering why would people take advantage of innocent users and their novice mistakes, I realised that’s how the world runs. You, as a user, have to supremely careful and alert. It’s equivalent to leaving your user id and password on a public park bench.
After the session was over, Will sent me an email explaining what had transpired, and that since it was unauthorised use, he has put the case up for waiver of charges. He also sent a link to a gem that scans your code for exposed sensitive information and prevents you from doing something stupidly expensive.
Called git-secrets and owned by AWS, this further protects users/students from future exposures while pushing code to Github. Git Secrets scans merges, commits, and commit messages for secrets and rejects prohibited regular expressions patterns from being posted. Here’s where to get Git Secrets.
https://github.com/awslabs/git-secrets
A few tense, on-the-edge, panic-stricken days later, I just received an email from AWS saying they have waived the charges of over $3000. I can not say how relieved I was when I received that. I wanted to buy everyone at AWS a beer.
So please do be careful, very, very careful the next time you push your code to Github. Check, double check, triple check before you commit anything. And install Git Secrets.