Forward incoming mail using Amazon Services and Lambda

In this post I’ll try to explain how I used Amazon to forward our incoming mail to an e-mail address like gmail.

Update 25 jan 2017: I seem to have missed some comments on this post, because they ended up in my spam folder :-(. Personally I don’t use my own solution as described here anymore. I think we switched to namecheaps forwarding service after all. Anyways, I couldn’t help myself but googling a bit on this subject, and it seems somebody else came up with the idea of using lambda to forward e-mail. I haven’t tried it but this project has nice little green icons that says it has been tested, so it must be better than what I did here.

Update 16 feb: Updated the lambda script thanks to Chris in the comments :-)

Next day update: Forwarding mime coded mails didn’t show up too well in my mail client. Of course I was testing with plaintext first ;-). Also I found this github page, describing pretty much the same as this post, although they still use S3 objects which I find redundant. I updated my Lambda script. It now sends a raw message, copying the Content-Type and Content-Transfer-Encoding headers from the original headers. This seems to display much better :-)

Use case

Say you have a domain specifically for e-mail campaigns. To setup this domain as a whitelabel domain in Sendgrid, or whatever third party you use, you must probably verify the send address. To do so, you must be able to receive mail.

Maintaining a mailserver is a pain in the ass though, with all those evil spammers around. So we happily leave that to companies who are must better in that. On the other hand, we don’t want to pay for each campaign just to receive this stupid verification link. Forwarding e-mail seems like a nice solution.

However, we found that the services at namecheap, which included e-mail forwarding, isn’t always that reliable. So we switched to Amazon for our DNS services. Wouldn’t it be nice to use Amazon to receive and handle our e-mail as well? Could Amazon simply forward e-mail?

Well, turns out it can forward e-mail, but not with a one-click install. Here’s how I did it.

Services

I used SES for incoming e-mail (and eventually outgoing as well). SES pushes the mail to SNS. And SNS pushes the notification with a Lamda function to e-mail with a little javascript.

I found this post on the amazon forum, but this guy used a s3 bucket to pull the e-mail from. I don’t want to store the mail in a bucket; I want to forward it immediately.

So I changed the script to fetch the information from the SES message directly. After I had fixed the scripts, permissions, etc it turned out that SES doesn’t send the actual contents of the e-mail message to the lambda script. But if you push the message to SNS, then the SNS message does contain the contents. So therefore I push to SNS, and let SNS push to lambda.

Setup SES

SES is Amazons Simpel Email Service. It is able to send and receive mail. But you need to verify your domain first, as well as the Email addresses you are going to use. At first I thought I only needed to verify the e-mail address that I use in the FROM field, but if you are in sandbox mode, you will also need to add the TO address.

Just add it exactly the same way, by pushing the “Verify a new email address” button, and enter the e-mail address that will be receiving the forwarded mails. Of course you can also have your SES account un-sandboxed, but actually I like this extra layer of security.

After you verify your e-mail addresses, you have to setup the Email receiving part. You have to make sure that your domain is configured correctly to let Amazon receive the mail. If you use route53 (which I think you should), Amazon can configure this automatically for you.

When that’s all done, you can create a rule set to receive the mails. But first, we need to setup the lambda function, and then the SNS service.

Setup Lambda

The lambda function can be created in python, node.js and java. I already had a node.js script to start with, so I changed that a bit to fit my needs. Just go to the lambda section and follow directions on the screen until the part where you can ‘Create a Lambda function’.

In my case it first came up with a screen ‘Select blueprint’, but you can just skip that. Then fill in the name, description and runtime (Node.js). And then use this script, but change it a bit to fit your needs.

At handler I used index.handler, and at Role I used the basic execution role. You will need this role later on to add permissions.

Now you could run a test with SNS, but the example test data doesn’t contain an e-mail message, so this won’t work very well. It should not come up with any errors though.

The downside of this forward is that you cannot see the original “to” address in your e-mail client directly. It is sent in the X-Original-To header though, so you could use this to build a filter or something like that.

Setup SNS

Now go to the SNS service to create a Simple Notification Service. First you create a topic that you name ‘ForwardEmail’ or whatever you like.

Then on the navigation on the left you can click on “topics” which gives you an overview of the topics. Click on that geeky looking ARN name. Now you should be able to create a subscription for this topic.

Here you select AWS Lambda as protocol, and then select the endpoint you created in the previous step.

So now you have a notification service that forwards notifications to the lambda script. You should note that the lambda function is only suitable for e-mails though, so if you connect another Amazon service to this SNS things will probably break.

Setup SES again

Now back to the SES configuration again. Make sure both the forwardFrom and forwardTo address are verified at the “Email addresses” section.

Now create the rule set at Email Receiving.

I setup the first action as SNS, selecting my ForwardEmail SNS. Then as second action I added Stop Rule Set, but I’m note sure if this is mandatory.

Test your setup

When I was testing my Lambda script at the Lambda service section, I basically had two problems, after I weeded out all the bugs in my script. First I got an e-mail verification error, because I didn’t know I had to verify the receiving e-mail address as well.

Second, I received an error, something like this:

User arn:aws:sts::167718239101:assumed-role/lambda_basic_execution/awslambda_724_20160203104053770' is not authorized to perform ses:SendEmail' on resource `arn:aws:ses:us-west-2:167718239101:identity/forwarder@email-service-support.com'

Turns out that Amazon also has a very sophisticated permission management thing. To fix this, go to IAM under Services (Security & Identity). At the dashboard you can see that you have Roles, my dashboard says Roles: 1. Click on it and you will see an overview of your roles. Here you will also see lambda_basic_execution.

Click on it, and at Inline policies click on “Create Role Policy”. Then add this policy:

That’s it. At the service CloudWatch you should be able to see the logging of your lambda script. Also you should probably reduce the retention, because if you keep your logs forever I think Amazon will make you pay for it.

Hope this helps you. I am no Amazon expert whatsoever, so I’m afraid this is all I can tell you at the moment :-).

Samsung Xpress M2022W fix wifi problems

My mom was using her samsung m2022w printer for about 17 months when all of the sudden it just didn’t work anymore. I spent hours and hours trying to fix this and eventually succeeded.

Samsung xpress m2022w

Things I tried:

  • Update the firmware to it’s latest version. Be aware that you cannot change it back easily. I could not find older firmware versions on the samsung website.
  • Update the software of course.
  • Changed the location of the printer, because it said the signal was poor.
  • I took it home, because the router also recently changed, so I figured the new modem from the internet provider was just crappy.

In the end, as a last resort, I tried configuring the IP static one more time. I disabled the “auto IP” option, left it at DHCP (instead of BOOTP), and choose to enter my IP manually. Filled in all the details, and this time I got lucky and it connected to the network without any issues. By the way, to do this I had to connect the printer with the USB cable. Also disable Wi-Fi Direct because I don’t think the printer has two wlan radio’s on board.

I first changed these settings in the Easy Print Manager. The wifi wizard wasn’t working there, Windows would give an error telling me I should install the right software or something like that. But I was able to configure the network to be manually. Although it didn’t really feel like it was saving the configuration, but I checked by turning the printer off and on again. The settings were still there. Also now when I turned the printer on, the WPS button turned blue, without flashing, and stayed blue.

Now I could run the diagnostic tool. Here I updated all drivers, but on the bottom there is also an easy wifi setup tool thingy. At the version I used, I could also change the TCP/IP settings here. Just to be sure, I also configured the IP as static in here. To be sure it didn’t conflict, I used another static IP though, because I wasn’t sure which config would win: the one from Easy Print Manager, or the one using this wizard kind of setup.

I had used this wizard many times before, but when it was sending my configuration to the printer, that would just fail. This time it didn’t fail, it went right through, and I could finish the setup.

Finally I could also disconnect the USB. Opening up Easy Print Manager now showed the printer with it’s static IP, and I could print from notepad. Great!

So bottom line: use a static IP.

Update 29 jan 2017: In the end, we returned this printer twice to be repaired. On the one hand it’s great that Samsung gives this kind of support on such a cheap printer. On the other hand it’s really disappointing that we had to go through all this… packaging, waiting, etc. This first time they only updated the firmware which didn’t work. Second time they actually replaced something (can’t remember what though) and I think it works better now.

HP Laserjet 400 slow job processing

At the office, we got ourselves a nice HP Laserjet 400. Because it could do all things: print, fax, scan, and it should be OS X compatible too.

1329315003

But when we wanted to print a document from our laptops, all connected to our gigabit LAN, it took ages before the printer finally started processing the job. So I installed some additional memory to the printer, which wasn’t too expensive anyways from eBay. Still no improvement. Some prints would come out fast, some prints didn’t. I thought it had something to do with processing the document to the right format for the printer. Isn’t that what PCL isn’t all about? Maybe the OS X drivers couldn’t talk to the laserjet in the most efficient way. Maybe the drivers aren’t that good after all.

Eventually I kind of gave up on the machine. My colleagues were already using our old laserjet 1320 anyways. The Laserjet 400 basically degraded to a really expensive scanner, at which it’s not very good either by the way.

Until one day extra people joined our team, and we moved the furniture around a bit in the office to create extra workspaces. This time I connected the printer through wifi, and now it seems that prints are coming through way faster.

So although I don’t understand why this is working better, just give it a try. It might work for you as well!

Midnight Commander os x terminal shift keys fix

Of course you are a command-line guru like me, but once in a while it’s just so much easier to use Midnight Commander. Especially when renaming files or directories, where you can just edit the name windows-style. Well, if your shift key works like it should that is.

I couldn’t bare that it wasn’t, so I did some digging and found out that os x term sends the wrong codes. I found this solution on superuser, but what to do with the other shift keys? I also found this article that looks outdated, but is still pretty cool. I ended up mapping F13 to shift F1, F14 to shift F2, etc. using the X11R6 xterm codes. That’s because in this bug report somebody said that SHIFT F6 should output ^[[17;2~ so that matches the xterm codes.

Eventually I mapped all my SHIFT F-keys (which I don’t use much anyways) like this.

OS X term keyboard mapping for Midnight Commander

jQuery Validation form valid event

I was using the jQuery Validation plugin in a project, but I couldn’t find a proper option that I could use to find out if the form indeed was completely valid. We wanted to show a message like “You’re almost done, just hit signup now and you can start right away!” when all fields in the form were valid. I guess it’s the call to action stuff that marketing guys really like.

Using a $(form).valid() inside the unhighlight would cause a too much recursion error, which actually makes sense. The form itself doesn’t seem to have an invalid state either, when I checked with firebug.

So in the end, all I could think of, was counting the number of valid inputs, and see if they were all valid. I must see, I only used input text and selectboxes. Beware that also hidden inputs will be counter if you use the input selector.

I left out all the other code, so, this is the piece of code that really matters.

You could also do it the opposite way, and count the error elements. I actually did this too, because another comforting text was required when the e-mail input was hit: don’t worry, we won’t sell your data! This actually seemed a bit harder because there were input field that didn’t have a error message. But, surprise surprise, the error element was placed anyways, it just didn’t contain any text.

Also, when you unhighlight, the element doesn’t go away, it just becomes invisible.

But well, if you want to fool around with it, this is the selector I came up with. Note that my error messages go into a feedback container, but I suppose you could just search on the class name as well.

It’s actually the :parent and :visible that does the trick. I wasn’t sure if I could combine them, but it seems to work. According to the jQuery document, :parent is the inverse of :empty.

So you could use this if you only want to show a messages when no more (visible) errors are there anymore, for example if you use the same space. This is a bit different than count the valid options, because if you have invalid options that don’t have an error message, and therefore don’t use space, it still may be nice to show some message.

WP All Import: calculate discounts using your own function

Using WP All Import, I wanted to use a discount on our prices, but do the math myself. This way I could round the discounts by x.95 or x.45 instead of other weird prices.

The manual suggests you put your function in the functions.php of your theme. But I don’t want the function to be gone when updating the theme, and neither want to create a child theme just to put this one stupid function in. So I thought I’d create a plugin.

This seems to work perfectly. Just put in this code in your /wp-contents/plugins directory. Save this code to, for example, wp-all-import-discount.php.

Now you can easily use it in your import config:

WP All Import discountAs second parameter you can use the discount you want. Use 0.9 for 90%, 0.5 for 50%, etc.

 

Fail2Ban on CentOs 6.6 with SELinux

So, I kind of manage a server for somebody else, running on CentOS and Plesk. Today I found out again why I hate this combo so much. The box auto-updates Plesk and other stuff, and all of the sudden my fail2ban wasn’t working anymore. I accidently found out because of all the password errors in de syslog.

Turns out there is a problem with SELinux. I have heard of that name before, but never knew what it was or what it did. Now I do, although I still don’t really understand. All I wanted was to get fail2ban running again.

So when you google for SELinux and fail2ban you get a lot of posts about this error. The suggestion I saw most was to run sudo /sbin/restorecon -v -R -F /sbin but for me this had no effect. The label was still bin_t after that.

Then I tried to create my own module. This didn’t succeed in the beginning, but after adding and adding stuff from the log, I finally got to the point where a restart of fail2ban didn’t give me errors. First I had to install audit2allow, because that wasn’t on my system.

Eventually, I came up with this. Save this under /root/myFail2ban.te :

Now, as root, compile this thing. Enter this while you are in /root:

make -f /usr/share/selinux/devel/Makefile

Output should be something like this:
[root@web ~]# make -f /usr/share/selinux/devel/Makefile
Compiling targeted myFail2ban module
/usr/bin/checkmodule:  loading policy configuration from tmp/myFail2ban.tmp
/usr/bin/checkmodule:  policy configuration loaded
/usr/bin/checkmodule:  writing binary representation (version 10) to tmp/myFail2ban.mod
Creating targeted myFail2ban.pp policy package
rm tmp/myFail2ban.mod.fc tmp/myFail2ban.mod

Now, enable the module:

semodule -i myFail2ban.pp

Restart fail2ban and check your fail2ban logs for errors, and your /var/log/audit/audit.log if ACL’s are still denied. This worked for me, with just the ssh jail active. Maybe if you enable other jails, extra acl’s are needed to those logfiles.

If it doesn’t work, you probably want to remove the module again. Just type:

semodule -r myFail2ban

You can list all active modules with semodule -l.

WP All Import cron from CLI (update)

My previous post about executing the WP All Import cron jobs with PHP cli didn’t work too well. Apparently the processing part still stops executing after a while. I ended up with import jobs stating “last activity 5 hours ago”.

Still I don’t want to run this through the webserver. So I wrote a little wrapper shell script to execute my imports in cron. It checks if the line “is not triggered” was partially returned. The wp cron script would return “<p>Import #26 is not triggered. Request skipped.</p>” if you execute the processing action without triggering it first. In other words: the import is really complete now.

I put the shell script outside my document root, and it uses the “logs” directory to store the log files. So make sure this directory is writable and points to the right path.

Now just put this in your /etc/cron.d/yoursite:

 

WooCommerce: remove empty SKU products

At work, we got ourselves a nice WooCommerce shop. Using WP All Import, it automagically imports the products and updates stock. But somehow in we ended up with products in our database with no SKU.

Removing one of those products, and then rerun the import, seemed to work. So maybe these products where in the product feed once without a SKU. Who knows. Anyways, I wanted them gone, and didn’t want to delete the entire database, afraid of what Google may think of that.

So I checked the database a bit and came up with a SQL query. First, to check if you have products with empty SKU’s as well, you could run something like this:

To remove the products, I think this should do. Of course this wouldn’t delete the files in your upload folder. If you want to delete those as well, you might want to write a script that selects the records, and then removes the images as well.

 

Forms in Symfony2: dependent selectboxes

My wish was simple: I wanted to have an extra dropdown box, to be able to filter a table, so the second dropdown box would have less items. So the second dropdown box depends on the first. I found two nice articles about this, but I missed a few things before I got it to work. This post tries to describe what my pitfalls were.

The great articles I found were: Symfony2.4: Dependent Forms and Symfony2 – Dynamic forms, an event-driven approach

The difference with my approach is: building the lists of the selectboxes should be inside the form builder. This way I think it’s more reusable and code is in one place. I don’t want to write extra methods in my controller to fill the selectboxes with javascript. My idea was just to submit the form, and let the form figure out what it should do: save the object, or fill the second list with options.

To describe my pitfall’s, I better first describe my situation. I changed the use case for the sake of this article, but it’s the same with my problem. Suppose you have a Person who can own several cars. On the person edit page, where you can edit his name and other properties, I wanted to add a table with the cars he owns. Under this table I have action buttons for New, Edit and Delete. When you use new, the div on that page is reloaded with the form, so the page isn’t reloaded, you are still on the edit Person page.

This would be my database model:

Forms in Symfony2: model used in this example

The list of cars with all their types would be huge, so I wanted to select the Brand (Opel, Mercedes, BMW) first. This wasn’t a value that I should save to my Person_has_Car model, so I set mapped to false. This was my main problem, because now this value wouldn’t be mapped to the entity, so how should I read it?

Initially I wanted to post the BrandId to the form, and based on this BrandId I wanted to build the select. I found that the PRE_SUBMIT and POST_SUBMIT weren’t called when you don’t submit the entire form. Probably because of CSFR that doesn’t match.

Then after a long search I found that at the PRE_SUBMIT Form Event, the data was just an array instead of an object.

Well, after a few days of frustration, googling, trying, googling and trying, this is what I came up with. Maybe it’s not the best approach, but it seems to work.

So Image that you are on a “User edit” page, and you want to add a car to his account.

Form

My form class PersonHasCarType would be something like this:

Twig template

The (simplified) twig form would look something like this:

The $.parseHtmlBlock is just a little code snippet that replaces HTML based on a json response. It also does some initialisation. The snippet looks like this:

Note the clearOnChange in the javascript that clears the second selectbox (if it exists) with empty values. If I don’t do this, the form has validation errors the second time you change the brand of the car. This was an easy hack to prevent this, and also I think it’s actually not that bad to reset invalid fields.

Controller

Now my controller is something like this:

So basically we now have a form that submits itself whenever the first selectbox changes. The controller checks if the form is fully entered by checking if the car object is set. This approach will probably conflict if you use validation. But for me this seems to work in my application.