Posts Tagged svn

Installation Script Using WordPress Subversion Repositories

Posted by pbg in General, Web Development, php on November 7th, 2009

I’d like to take the example of svn externals from my previous post a bit further. I created a bash script that captures a group of commands that one would need to deploy a vanilla WordPress website with a cherry picking of plugins and themes. This script is a big-ass time saver. It’s especially useful for deployment of websites based on a package like WordPress or Drupal, that host publicly available subversion repositories. I tested it recently and made two websites spending two hours on each site, four hours total.

Web developers as they learn start by downloading packages in zip and tar.gz archives which is fine. A pro technique is to pull in the resource via SubVersion (svn). Svn provides additional features such as svn externals and svn hooks. Svn externals allows for the pulling in of external resources into a defined project in a svn repository. Svn hooks is for additional labor saving scripting that allows for the automation of repeated tasks such as a script that can pass along data from commit messages into other resources in the project such as the bug tracker. Learning these tricks allows for labor savings at every step in the production cycle. We can take these practices to the next level with capturing the commands in a bash script and then organise all the features of a website in a deployment script, something that is done all the time at software majors.

This script is a big-ass time saver.

I took the time to sort this out recently because I had some web development requests from people who have simple websites that could be converted to WordPress easily enough, but I wasn’t interested in going fishing for plugins and themes twice, and again when the next people come along needing the same thing. The job of a software developer should be to automate processes. I say often that the credo of developers is not to work for a living, but to eliminate work. But this idea is not always employed by developers in all the places it could, and its sometimes even a harder to get a client on board to make full use of methods and procedures that automate tasks and eliminate work.

…the credo of (software) developers is not to work for a living, but to eliminate work.

Lets at least cover the benefits in point form:

  • The script takes a couple of minutes and you save hours.
  • You dont repeat the labor, but you can repeat the use of the script, deploying anywhere else.
  • All your stuff is there at the beginning. Useful for planning, development, and policy across teams.
  • You are organised, and you can develop variations. Svn export may be good enough for your needs.

In pseudo code, here is what your script will do:

  • Set up your repository.
  • Make your directory structure for your project.
  • Check out your repository.
  • Run procedures for svn externals for core WordPress.
  • Run procedures for svn externals for plugins, iterating through data in an external file resource.
  • Run procedures for svn externals for themes, iterating through data in an external file resource.
  • Grab additional resources in an array, iterate through them using wget command, extract them.
  • Cleanup.
  • Commit message.
  • … anything else you can think to do.

The bash file, save as getallwpsvn.sh:

#!/bin/bash
# run this script with chmod 755 permissions.

workPath=$(pwd)

rm -rf filerepository repository www *.zip # this line cleans dir for testing, comment out when done

svnadmin create repository

mkdir -p filerepository/{branches,tags,trunk/{html,db,cron,scripts,themes,plugins,project,selenium}}
# got anything to import into those directories under trunk?
# import into the directories under trunk now
# before the next step
svn import filerepository file://$workPath/repository -m "initial import using getallwpsvn.sh script"
rm -rf filerepository
svn checkout file://$workPath/repository/trunk www
cd www
svn rm html
svn commit -m "rm html temporarily for clean propset"
svn propset svn:externals 'html http://core.svn.wordpress.org/trunk/' .
svn up
cd html/wp-content/
# get plugins from repository http://svn.wp-plugins.org/
# plugins listed in svn.plugins.externals
svn propset svn:externals -F ../../../svn.plugins.externals plugins/
#svn commit "plugins propset" # no commit if no local repository
svn up
# themes repository: http://svn.wp-themes.org/
# themes repository is a bit of a ghost town, none grabbed here
# browse the site and get the zip
# themes listed in svn.themes.externals file, if there are any
svn propset svn:externals -F svn.themes.externals plugins/
svn up

cd themes
# load up on themes
#more human readable format for array

THEMESITES[0]=http://dev.digitalnature.ro/fusion/fusion-wordpress.zip
THEMESITES[1]=http://ericulous.com/?load=googlechrome.zip
THEMESITES[2]=http://ericulous.com/?load=internetcenter.zip
THEMESITES[3]=http://ericulous.com/?load=redbusiness.zip
THEMESITES[4]=http://wordpress.org/extend/themes/download/elegant-box.4.1.1.zip
THEMESITES[5]=http://wordpress.org/extend/themes/download/thirtyseventyeight.4.0.zip
THEMESITES[6]=http://wordpress.org/extend/themes/download/thirtyseventyeight.4.0.zip
THEMESITES[7]=http://wordpress.org/extend/themes/download/constructor.0.6.4.zip
THEMESITES[8]=http://wordpress.org/extend/themes/download/jq.2.4.zip
THEMESITES[9]=http://wordpress.org/extend/themes/download/ahimsa.3.0.zip
THEMESITES[10]=http://wordpress.org/extend/themes/download/retromania.1.3.zip
THEMESITES[11]=http://wordpress.org/extend/themes/download/skinbu.1.0.3.zip
THEMESITES[12]=http://wordpress.org/extend/themes/download/mystique.1.16.zip
THEMESITES[13]=http://wordpress.org/extend/themes/download/lightword.1.9.3.zip
THEMESITES[14]=http://wordpress.org/extend/themes/download/monochrome.2.3.zip
THEMESITES[15]=http://wordpress.org/extend/themes/download/thematic.0.9.5.1.zip
THEMESITES[16]=http://wordpress.org/extend/themes/download/hybrid.0.6.1.zip
THEMESITES[17]=http://wordpress.org/extend/themes/download/new-york.1.0.1.zip
THEMESITES[18]=http://wordpress.org/extend/themes/download/f8-lite.1.3.zip
THEMESITES[19]=http://wordpress.org/extend/themes/download/simplex.1.3.1.zip
THEMESITES[20]=http://wordpress.org/extend/themes/download/cleanr.0.1.2.zip

for s in ${THEMESITES[@]}
do wget "$s"
done

FILES="*.zip"
for f in "$FILES"
do unzip "$f"
done

rm *.zip
rm *.zip.*
cd ../../../
svn commit -m "load in of plugins and themes complete"

cd $workPath
cp $workPath/www/html/wp-config-sample.php  $workPath/www/html/wp-config.php
chmod 777 $workPath/www/html/wp-config.php
chmod 777 $workPath/www/html/wp-content #temporarily, for cache
mkdir $workPath/www/html/wp-content/uploads && chmod 777 $_
touch $workPath/www/html/.htaccess && chmod 777 $_

# do any post processing, other importing now, and commit it if you did.

Set the file permission to chmod 755, and run it from the shell command line as in ./getallwpsvn.sh.

The file you save as svn.plugins.externals:

all-in-one-seo-pack http://svn.wp-plugins.org/all-in-one-seo-pack/trunk
advertising-manager http://svn.wp-plugins.org/advertising-manager/trunk
cforms http://svn.wp-plugins.org/cforms/trunk
google-sitemap-generator http://svn.wp-plugins.org/google-sitemap-generator/trunk
sociable http://svn.wp-plugins.org/sociable/trunk
stats  http://svn.wp-plugins.org/stats/trunk
ultimate-google-analytics http://svn.wp-plugins.org/ultimate-google-analytics/trunk
vipers-video-quicktags http://svn.wp-plugins.org/vipers-video-quicktags/trunk
wordbook http://svn.wp-plugins.org/wordbook/trunk
wp-flickr http://svn.wp-plugins.org/wp-flickr/trunk
wp-super-cache http://svn.wp-plugins.org/wp-super-cache/trunk

The svn.plugins.externals file is a name – resource listing, one per line, when you have more than one resource to define with svn externals.

Please note that you may not need all of this; comment whatever out you want. You dont need to create a local repository, that is only if you are doing team development, or perhaps custom development on themes and plugins. I found though that it was necessary with svn propset directives to create a top-level directory structure wherein is stored all the different directories. The point of this exercise is a pull-in of public resources in a step that you can repeat automatically over and over. It also need not be a very sophisticated script to get the benefits from it.

, , , ,

No Comments

The Power of svn externals

Posted by pbg in General, Web Development, php on October 17th, 2009

I have a list of svn tricks over on a previous post, (my) Essential Cheat Sheet of Shell Commands. But I was listening to a very lengthy, and I mean really lengthy Zend podcast, over two hours in length: The ZendCon Sessions Episode 26: Best Practices of PHP Development, with Matthew Weier O’Phinney and Mike Naberezny. I learned a new trick about subversion I would like to share and note here for the future, svn externals .

I had of course heard of svn externals before but a little explanation really enlightened me. You can multiply your power as a developer with svn externals. Alongside your own project under svn control, you can add in other remote projects from remote svn repositories with the svn propset command and they will naturally remain current as you run updates on your own repository.

You can multiply your power as a developer with svn externals.

In the struggle for project housekeeping it can be a chore to keep libraries, plugins, middleware, and other goodies up to date. In general we as developers have a mindset about keeping a project managed by version control, but its an idea that I have seen stopping at the project in question. The project is under svn, the rest of the libraries are from static resources like tarballs. By just extending the power of version control just a little bit further, we have a big labor saver, and we are opened up to the great universe of software. Svn externals gives us mighty lever, because we then have the power to keep in step with all of the other bits and pieces that go along with a project.

Your main project is under version control, but your rich html editor, TinyMCE, is not. Well it can be. Lets use it as an example.

test$ mkdir tmc
test$ cd $_
test/tmc$ mkdir html
test/tmc$ mkdir project
test/tmc$ cd $_
test/tmc/project$ mkdir branches tags trunk
test/tmc/project$ cd -
/home/pbg/websites/test/tmc
test/tmc$ cd html/
test/tmc/html$ ls
test/tmc/html$ emacs index.php
test/tmc/html$ cd ../
test/tmc$ cp -rf html project/trunk/
test/tmc$ ls
html  project
test/tmc$ ls project/trunk/
html
test/tmc$ ls project/trunk/html/
index.php  tiny
test/tmc$ ls
html  project
test/tmc$ rm -rf html
test/tmc$ ls
project
test/tmc$ svnadmin create tmcrepository
test/tmc$ svn import project file:///home/pbg/websites/test/tmc/tmcrepository -m "initial import"
Adding         project/trunk
Adding         project/trunk/html
Adding         project/trunk/html/index.php
Adding         project/branches
Adding         project/tags
Committed revision 1.
test/tmc$ ls
project  tmcrepository
test/tmc$ svn checkout file:///home/pbg/websites/test/tmc/tmcrepository/trunk .
A    html
A    html/tiny
test/tmc$
/test/tmc/html$ svn propset svn:externals 'tinymce https://tinymce.svn.sourceforge.net/svnroot/tinymce/tinymce/trunk' .
/test/tmc/html$    svn commit -m "propset"
/test/tmc/html$    svn up
/test/tmc/html$

So that is how its done from bash with a vanilla website and one repository checked in for your pleasure. Tips to know include wrapping the directory and resource in quotes, don’t create the directory, specify it in propset and let svn create it for you. Go to a directory somewhere else on your system and test your checkout. In the example above the remote repository trunk is checked out. However, you are also able to check out specific branches or even specific revisions if you want the bias more to stability over new features. You can also specify multiple remote repositories by creating a text file with directory and remote resource pairs and point svn propset at that file. Be prepared to handle things that you flub up using svn propedit. Your repository is not broken, but you may have to know how to fix a thing or two that you didn’t do the first time. That is why doing a vanilla procedure like what is described above helps show what the right way is.

So Imagine having a repository somewhere with all the tools you like to work with as part of your own best practices, sitting there, in one place ready to go with one checkout. That whole kit and kaboodle becomes your blank slate from where to start from, but you are already miles ahead of the competition because you already have tabs on all the resources you are going to use in your project. Having it all on hand saves labor and helps foster better practices as a developer.

  • Zend Framework, CakePHP, Symphony, PEAR or whatever middleware turns your crank
  • PHP Unit, or SimpleUnit, or some other unit testing suite
  • Wordpress, Drupal, Joomla, or of course any of the related themes and plugins
  • Tiny MCE
  • Jquery, Scriptaculous, Dojo, whatever floats your boat with javascript
  • XDebug,
  • phpmyadmin
  • Integration testing software like Selenium or Molybdenum
  • bloody well anything from sourceForge, or anything else for that matter public and under svn control.

Links for this blog post:

http://www.superwebdeveloper.com/2008/10/08/essential-cheat-sheet-of-shell-commands/

http://devzone.zend.com/article/9930-The-ZendCon-Sessions-Episode-26-Best-Practices-of-PHP-Development

http://beerpla.net/2009/06/20/how-to-properly-set-svn-svnexternals-property-in-svn-command-line/

http://blogs.gnome.org/johannes/2008/02/20/svnexternals-for-noobs/

, ,

No Comments

Essential Cheat Sheet of Shell Commands

Posted by pbg in GNU Linux, General, Web Development on October 8th, 2008

Though I would add a list of shell commands I keep using as a guide to myself and others.

svn
make svn code directory, call it codebase. Put a trunk, branches, and tags directory below it. Import all that codebase under trunk. That means, the whole site. Also, make a directory for your db, and put a db dump in there. If you are running a cron, make a directory called cron, put your cron scripts in there and a text file copy of your crontab. Don’t store passwords in subversion, as in the top level file that your site uses. Make a version of the file without the passwords, call it something like config.orig.php or whatever, and check that in instead. Also, you may need the equivalent of the CVS ignore command called svn propset and make use of it.
#svnadmin create repository_directory

#svn import codebase file:///home/user/pathtorepositorydir/repository_directory -m “initial import”
check out your remote repository into your local machine at the command line:
#svn checkout svn+ssh://user@domain.com/home/user/repository_directory/trunk .
local:
#svn checkout file:///home/user/pathtorepository/repository_directory html/

now you are checked in and out, you can delete codebase directory.

svn export works in a similar way. It pulls out the files from the repository sans .svn directories.
#svn export svn+ssh://user@domain.com/home/user/repository_directory/trunk

grep
find instance and string in and below current directory, pipe it to less.
#grep -r “string” * | less

find
find all those old CVS or .svn directories, and kill them:
to look:
#find . -type d -name “.svn”
to dump to a file
#find . -type d -name “.svn” > dump.txt
when you are ready:
#find . -type d -name “.svn” -exec rm -rf {} \;

Another way to wipe out everything:
#find . -name ‘*’ -print0 | xargs -0 rm
this means find here, the name of all, dont print it to stdout, then redirect the output to as an argument that the rm command will execute on.

scp
push a file
#scp localfile.txt user@domain.com:pathtofile/remotefile.txt
grab a file
#scp user@domain.com:pathtofile/remotefile.txt localfile.txt

rsync
get all those image files
#rsync -avz user@domain.com:/home/pathtofiles/ .
ah yes, but I will get a complaint  from subversion about my directory being out of sync because what I just did was downloaded the .svn file from the server over that directory. They arent the same, so your svn update now crashes. What do do?
# rsync -avz –exclude=.svn  …….then everything else after that. More options in the man pages.

mysql
Export a db
#mysqldump -uuser -ppassword -hlocalhost dbname > db.sql
Import a db
#mysql -uuser -ppassword -hlocalhost dbname < db.sql

crontab
A crontab line should point to a .sh script. The .sh script can then execute the shell script. This enables you to use either the sleep funciton or looping constructs to run the script a multiple of times if you like, and  keeps your command to one line inside the cron.
Look at your crontab with #crontab -l edit it with #crontab -e
Crontab time examples:|
1 */3 * * *  every 3 hours, one minute after the hour. Where possible, dont run a crontab exactly on the hour, because that is when everybody else does it on a shared host. Set it for a minute after when the cpu isnt likely to be so taxed.
1 0,12 * * * every 12 hours, one minute after the hour.
*/1 * * 3/6 every minute on every third and sixth day of the week

wget
use wget to grab that tarball directly, for when you are grabbing such things off sourceforge or wherever.
# wget http://domain.com/pathtofile.tar.gz

tar
good old tape archive.
To extract:
#tar -xvvf filename.tar.gz
To archive:
# tar -czvf tarballname.tar.gz directory

screen
Got a shell connection to a dodgy host that keeps giving you the boot? after you login, run screen:
#screen -DD -R

You will still get booted, but at least you can get right back to where you were when you reconnect by replaying the above command.

mysql
you probably know mysql command line access if you are on this page. You have to know how to work with this because a database can exceed the size allowable for transfer over http, making into phpMyAdmin impossible. But a goodie that I found is that case when you want to wipe out all the tables in a db, but not the db itself, in order to preserve all the privleges, and access credentials. The following line can save a step:
#mysqldump -uuser -ppassword –add-drop-table –no-data dbname | grep ^DROP | mysql -uuser -ppassword dbname

, , , , , , , , , , , , , , , ,

No Comments