Skip to content →

Month: May 2011

Run a Python program on every file in a directory

Use the cmd line to run a python program on every file in a directory

It took me several hours to get the command line right to run a python program on every file in directory and send the results to the same output file.

I am jotting down my notes here because it seems that much of the advice on the internet is plain wrong.  This is what worked for me

Assuming

  • We are working on a regular computer using Windows
  • Python is already installed
  • We have a Python program called myprog.py
  • We have data we want to analyse with that program
  • We have many, many data files in a directory and we want to analyze them one-by-one and put the results in one file.
  • The program file expects one or more data files as sys.argv. That is, the program expects us to tell it where to find the data files in the cmd line using this format.
    • Myprog.py sys.argv1 sysargv2 sysargv3  >output.txt

If we only have a few data files, or we are only using a few files in a directory, then we don’t need this procedure. It is easier simply to type out the command as shown above.

If we want to run the program on a thousand files, say, it would be tedious to type out the filenames (in place of sys.argv).

Gist of procedure to use cmd line to run a python program on every file in a directory

To run a python program on every file in directory, we use the for command.  Here are the steps in an orderly way.

Steps:

#1 Locate the directories of the python program and the data

For example

Python program physically sits in c:python27myprog.  The name of the program is progname.py.

Data physically sits on a removable drive e:mydata

#2 Decide where we are going to send the results

I always send my results to the same directory as the program but that is not necessary.  The reason I do that there is less chance of a type if I don’t have to type in the directory name and it is relatively easy to move a file to another directory manually.

In my example

My results will go to the same file as the directory c:python27myprog

#3 Open the cmd line in Windows

We are going to give commands directly to DOS, the operating system underneath Windows.  You may dimly remember it from 1980’s computers.  Yes – it still runs Windows machines.

  • Go to Start (bottom left – where you normally switch off your computer)
  • Select Run
  • Type cmd<enter> into the box : a black square should pop up.

#4 Set the directory of our program as our working directory

In the “command line window”, you should see something like c:mydocuments . .  Windows machines try to make us work in My Documents.   I change this to my program directory using

cd c:python27myprog

That means change directory to c:python27myprog

#5 Use the cmd line to tell python to run myprog iteratively with every file in the data directory

for %f in (e:mydata*.txt) do myprog.py %f >output.txt

Notes

  • Look at the syntax : for  variable in (set) do myprog.py variable >outputfilename
  • The variable name must be %f or %x or %t.  A percentage followed by one letter. Everything else throws up an error.  The variable name must be the same in both uses.
  • The directory holding the data is put in brackets : (dirname)
  • >outputfile.txt sends results to a textfile in the same directory as our program.  It must be a .txt file.

Conclusion: Use cmd line to run a python program on all the files in directory

And remember that if the program takes 1 seconds to run for one file, then for 1800 files, as I am running now,  then  the program takes 30 minutes to run.  If the original program takes 2 sec to run, the total batch will take an hour.

Leave a Comment

How to set up a new website – Steps 4 & 5 – Link your Google Apps to your website

Step 4 Fireworks by Jason O'Halloran via Flickr

Situation:  Google Apps wants to verify that I own my domain

  • Fair ask but if there is one truth in this world, Google staff can’t write for toffee. Prepare for confusion.
  • I got the impression that after I had activated my hosting, that the instructions on Google Apps changed.  But who knows – I found the instructions on Google Apps a tangled maze.
  • Google Apps offered me an html file to copy and put on my website.  So that is what we are going to do.  Particularly as I know now that words.

Task:  To load Google Apps verification code on my website

#1 Save the verification file in its html file on my PC

  • I also save it in a folder called mydomainname/websitemaintenace/this file must be on the website

#2 Go back to the hoster’s to load it up using their FTP service

I could use another FTP service but then I would be fiddling about with server names and passwords and that would be irritating

  1. Log back in to the hosters
  2. Go to Manage Domains
  3. Without rushing around and pressing the wrong thing, choose WebFTP
  4. Again, carefully, look at the file structure
  5. Position myself in the root folder for mydomainname.  I should see wp-content there among other basic WordPress files.
  6. Browse my PC and select the file that I have just saved containing the verification code.
  7. Upload (which in Dreamhost is an obscure confusing green arrow near the top of the page).
  8. Check the file is there.

Step 5  Fireworks by Jason O'Halloran via Flickr#3 Go back to Google Apps and confirm that I have loaded up the file

  • Make sure that I press Verify!

#4 Still in Google Apps, go to settings and choose all the simplified names

  • Mail.mydomainname.com
  • Docs.mydomainname.com
  • Calendar.mydomainname.com
  • Sites.mydomain.com

Whew.  Done

  • I have a domain name registered (best to put the renewal date in my calendar now and activate and automatic email)
  • I have my a WordPress shell on my hosting service
  • Also on my hosting service is a little piece of Google code so I can use Google Apps as my email service
  • I have Google Apps set up with simplified addresses.

Next

  • I can sign up all the users to Google Apps
  • Start sorting out WordPress at the hosters

Easy when you can find your notes. If you have any hassles, let me know.  I’ll help if I can.

One Comment

How to set up a new website – Step 3 – Set up your hosting and WordPress shell

Situation:  I want to make the outline of my website at my hosters

  • I’ve registered and paid for my domain name
  • The registrar has alerted me by email that it is ready for use
  • I have pointed my domain’s name servers to my hosting service and set the name servers on the hosting side (this time I did nothing because the hosting service were also the registrar and they did everything.)
  • While I was waiting, I registered a Google Apps account for email.

Step 3 Fireworks by Jason O'Halloran via FlickrTask:  To register for Google Apps

Now I want to tell the hosting service to host my site.  I will also confirm that I will be using Google Apps for email.  And I will set up a WordPress shell which I will use as the scaffolding for my website.

Steps:

#1 Log in to the hosting service

  • Find it via Google!
  • Find the login link – never obvious
  • Recall the email address and password used for login and ‘access to the cpanel’.
  • Look for “Manage domains’
  • Select the new domain
  • Make sure that the check boxes for [both http//www.domainname.com and http://domainname.com] and Google mail are ticked.
  • Have a good look at everything else [without touching].
  • Choose full hosting [this may be different for other hosters].
  • Double check by going back to Manage Domains. Does it seem that the hosters know they are hosting the domain?

#2 Now load up the WP shell

The reason for loading up WP now is that when you are asked to load it up, you are reminded to have no files in the ‘domain space’.  So I want WordPress in place before the link up to Google.  Maybe, one day I will experiment with doing it the other way.

  1. Go to one click install and choose WordPress
    1. Chose CUSTOM install despite your sense that you want a SIMPLE install.
    2. Choose your domain
    3. Enter your username and email as requested.   The WordPress shell will contact you via your email and you can complete WordPress stuff later.
  2. You can try typing http://yourdomainname.com in a browser and seeing if it comes up.  If it doesn’t, that could be because the name hasn’t propgated around the internet yet (meaning your internet service provider cannot convert a bland request of get http://yourdomainname.com into action because there is insufficient common knowledge to pass you along from service to service to your website.)

Progress you have made to launching your site

  • You know have a domain name registered and pointing at a hosting service
  • You have told the hosting service to host this domain name (to make space on their servers)
  • You’ve told it to recognize your name with or without www
  • You’ve told it to will be using Google Apps for email
  • You’ve loaded up a (CUSTOM) install of WordPress and your hoster has done the few additional tasks such as make your database and install the WP software.  You can fix up your WordPress later.

Next – fetch code from Google

And upload it to your website so Google is authorized to run your email

Leave a Comment

How to set up a new website – Step 2 – Sign up for Google Apps

Step 2 Fireworks by Jason O'Halloran via FlickrSituation:  I want Google Apps to give me gmail with the same name as my website

  • I have bought a domain name from a registrar and my domain name  servers are pointing at my hoster (actually I bought my name through my hoster so the domain name servers were already pointing in the right direction)
  • I could use my hoster’s email but they encourage use to use gmail and I want the other benefits of a virtual office that I get through Google Apps
  • I don’t want a gmail name.  I want to use my name.

Task:  To register for Google Apps

Steps:

#1 Find  Google Apps

#2  Note that Google Apps is telling you to verify your domain

  • I got sent hither-and-thither
  • Relax – it works out but it is spaghetti
  • Leave this all alone for now and wait for your domain name to be registered.  You will come back when this is done.


The step you have taken to launch a new website with email

  • You have registered a Google Apps account so you can use gmail with your domain name
  • You have to go back to your hosting service to set up your shell ready to link up to Google Apps
Leave a Comment

How to set up a new website – Step 1 – Get a domain name

Step 1 Fireworks by Jason O'Halloran via FlickrSituation:  I want to set up a new website

  • I’ve already mocked up what I want on a local (WAMP) server on my PC (not necessary to complete this step but I have done).
  • I’ve thought up a domain name which I am happy with.
  • I’ve already checked on Domainr, or a similar service, that my name has not been taken (is available.)

Task:  To buy a domain name

And have its nameservers pointing  at my hosting service, i.e. so customers who ask the internet for my domain name are set to the computer where my website will sit (be served).

Steps to get a new domain up-and-running

#1  Registrar

First, I must choose a registrar where I buy the domain name and renew it annually.  A registrar is a telephone directory or index for internet.  There are many and they cleverly cross-reference each other.   The best know is GoDaddy.com.

To make my choice

  1. Does the Registrar cover the address that I want?  A Registrar covers a  limited range of TLD or top level domains.  For example, a registrar in the US may not cover .co.uk addresses.
  2. How easy is it to point the Name Servers that will be listed with my domain name to the computers that will be hosting my website?  Have I got clear instructions from both ends – the registrar service and the hosting service (or Posterous or WordPress.com – the place where my website will physically sit)?
  3. If I have more than one registrar on my shortlist, is there any difference in their prices and reputation?
  4. Do I want to use my hosting service as my registrar or is it better to have ‘two suppliers’?
  1. My hosting service might give me a discount on the domain name (a few pounds or dollars) but now they have more power over me (they have my site and my domain name under their administrative control).
  2. I may have to buy my domain name elsewhere if they don’t offer my preferred TLD (top level domain – like .co.uk).
  3. If I buy (and renew) my domain name with a separate registrar to limit the power of your hosting service over me but then I must remember to  my domain name on time and to pointing the name servers listed at the registrar to the IP address of the hosters where my  website physically sits.
    1. I must enter the data of the registrar at your hoster and your hoster at the Registrar!
    2. Adminstratively, I must have two sets of commercial transactions that I must diarize 1-2 years ahead and coordinate.

#2 Buy your domain name

Now I have chosen my name and my registrar, I must buy my domain name.  I chose to buy a domain name through my hosters.  That means I don’t have an additional task of pointing the name servers to them.   If you choose to split the hosting and the registration, you will need other instructions.

  1. Get the right credit card (business or personal)
  2. Go to the online home of the Registrar (and probably set up an account.  I used my hosting service and I already had an account.)
  3. Find the right page and click whatever button to buy a domain name.
  4. The registrar tests whether the name is available. (If they don’t, clear out fast!).
  5. When they have confirmed the name is available (a few seconds), they ask for credit card details and an email address.
  6. They also suggest that I list my address at their office rather than display my full address on the internet.  I don’t know the pros and cons but I chose to list their address because I am tired of spam.
  7. When money has changed hands, they promise me an email and tell me to patient.  It takes a day or two for the network of domain name servers to gossip among themselves that my domain name isnow taken and that anyone who asks for it should visit my hosters.
  8. Finally, a job is not finished ‘until the money is in the bank.’  Print two copies of their email and put one in the expenses file for the accountant and one in the file holding all the details about this website.

#3 Test the domain name

Type in the new domain name to the browser bar (not Google – the browser bar) and see if it comes up.

This one worked quickly but don’t panic for up to three days.

PS The name does not show up in Google and should not show up in Google. Their search spiders don’t know the site is there and we don’t want to be found yet.  There is nothing to see.

Leave a Comment

5 steps for setting up a new website

Fireworks by Jason O'Halloran via FlickrMy notes for setting up a new website

I am just in the process of setting up a new website and because I don’t do this often, I am writing down how to do it.

Nothing here is new.  What I write is not necessarily accurate.

These are my notes so that next time I don’t have to start from scratch.

Use my notes if they help you.  But read them intelligently knowing that I am not an expert.

Stages

  1. Get a domain name
  2. Sign up for Google Apps
  3. Set up your hosting & load up your WordPress shell
  4. Link up Google Apps to your hosting service
  5. Complete the link up at Google Apps

These steps take us from

  • I have found the name I want and I have chosen my hosting service (where my website sits physically).

To

  • I own the name.  Its “nameservers” point to my hosting service’s computers (where the website is physically)

And

  • I have a matching account at Google Apps for email with a piece of their code on my website on the hosting service computer ‘authorizing’ Google Apps to run this parallel service using the same name as my website
  • Note that if you don’t need a new email address then you don’t need steps 2, 3 and 5.

 

Leave a Comment